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ABSTRACT 



As policymakers have sought to balance the goal of fostering 
poor children's well-being with that of encouraging adult's self-sufficiency, 
public assistance has become more predicated on custodial parents ' 
involvement in work or mandatory welfare- to-work programs activities. This 
report examines the effects of welf are-to-work programs on the children of 
the adults mandated to participate. The report synthesizes the findings from 
two recently completed reports from the National Evaluation of 
Welf are-to-Work Strategies (NEWWS Evaluation) , the 2-year effects of 11 
welfare- to-work programs that operated in 7 sites in the early to mid 1990s. 
Section 1 of the report summarizes the findings. Section 2 presents a 
conceptual model of how mandatory welf are- to-work programs might affect 
children. Section 3 describes aspects of child well-being examined in the 
NEWWS evaluation. Section 4 discusses characteristics of the adults and 
children in the evaluation samples. Sections 5, 6, and 7 summarize program 
implementation and program effects on targeted and nontargeted outcomes, 
highlighting any situations where effects were different for mothers in a 
special Child Outcomes Study sample focusing on young children, compared to 
mothers with children of all ages. Section 8 presents effects on children. 

The report notes that, overall, effects on children were consistently 
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favorable in the cognitive development area, consistently unfavorable in the 
health area, and both favorable and unfavorable in the behavioral and 
emotional adjustment area. Child effects were not systematically different 
for mothers subject to employment -focused programs than for those subject to 
education- focused programs. Few child effects were found for subgroups of 
young children identified as at high or low risk for poor development. The 
report concludes by asserting that mandatory welfare- to- work programs, with 
no services provided directly to children, can have spillover effects on 
children's well-being. The report's four appendices include definitions of 
the child outcome measures and a comparison of national samples of children 
and control group children. Contains 19 references. (KB) 
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Introduction 



Since its inception the primary goal of the Aid to Families with Dependent Children 
(AFDC) program, as well as successor programs funded under Temporary Assistance for Needy 
Families (TANF), has been to provide government support for poor children. Over the years, this 
public assistance has become more and more predicated on custodial parents’ involvement in 
work or mandatory welfare-to-work program activities, as policymakers have sought to balance 
the goal of fostering poor children’s well-being with that of encouraging adults’ self-sufficiency. 
Currently, there are strong incentives for states to run mandatory, work-focused welfare-to-work 
programs: States face financial penalties if they fail to meet TANF-defined participation stan- 
dards, which require large proportions of welfare recipients to be working or in work-related ac- 
tivities, and states must require recipients to work after two years of assistance. In addition, fed- 
eral funds now may not be used to support most families on welfare for longer than five years, 
and a number of states and localities have shorter welfare time limits. 

This document examines the effects of welfare-to-work programs on the children of the 
adults (almost all single mothers) mandated to participate in such programs. Synthesizing the re- 
sults from two recently completed reports from a large-scale evaluation — the National Evalua- 
tion of Welfare-to-Work Strategies (^WWS Evaluation) — the two-year effects of 11 welfare- 
to-work programs that operated in seven sites in the early to mid 1990s are summarized.' The 
sites included in the evaluation are Atlanta, Georgia; Columbus, Ohio; Detroit and Grand Rapids, 
Michigan; Oklahoma City, Oklahoma; Portland, Oregon; and Riverside, California. While the 
programs operated under the federal Job Opportunities and Basic Skills Training (JOBS) pro- 
gram that preceded TANF, and thus did not invoke a time limit on eligibility for welfare, they 
shared TANF’s primary goal of moving welfare recipients into paid work and off assistance, and 
they reflect a range of approaches, implementation features, and environments: Some were 
strongly employment-focused while others emphasized basic education; they varied in how 
broadly the program participation mandate was applied to the welfare caseload and how strictly it 
was enforced, in the amount of child care support provided for program participation or em- 



'This synthesis is one of many papers and reports that have been issued as part of the NEWWS Evaluation, 
conducted by the Manpower Demonstration Research Corporation (MDRC) under contract to the U.S. Department 
of Health and Human Services (HHS), with support from the U.S. Department of Education. Child Trends, as a sub- 
contractor, is conducting the analysis of outcomes for preschool-age children (the Child Outcomes Study). This 
document synthesizes results presented in two 2000 reports, both published by the U.S. Department of Health and 
Human Services (Administration for Children and Families and Office of the Assistant Secretary for Planning and 
Evaluation) and the U.S. Department of Education: Impacts on Young Children and Their Families Two Years After 
Enrollment: Findings from the Child Outcomes Study, prepared by Sharon M. McGroder, Martha J. Zaslow, Kristin 
A. Moore, and Suzanne M. LeMenestrel (Child Trends); and Evaluating Alternative Welfare-to-Work Approaches: 
Two-Year Impacts for Eleven Programs, prepared by Stephen Freedman, Darnel Friedlander, Gayle Hamilton, 
JoAnn Rock, Marisa Mitchell, Jodi Nudelman, Amanda Schweder, and Laura Storto (MDRC). This synthesis is 
very much indebted to the authors of these reports, who conducted the many analyses included here. In addition, 
certain concepts and language in this document reflect the joint efforts of the two research teams. Note, however, 
that because of differences in the data sources used for economic impacts and differences in the definition of the 
sample of children examined, which are explained in detail later in this document, some statements regarding eco- 
nomic inqiacts and young child impacts may differ somewhat between this synthesis and the above-cited report on 
impacts on young children. 
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ployment, and in methods of case management; and the programs served different welfare popu- 
lations and operated in a variety of labor markets. Although the NEWWS evaluation was de- 
signed to address the effects on children of requiring parents to participate in welfare-to-work 
programs, there are many other policies — for example, child care and health insurance policies 
— that can affect children, and those policies can be examined only indirectly in this evaluation. 

To determine program effects on children, the NEWWS Evaluation uses a very strong 
research design: a random assignment experiment. In each evaluation site, adults who were re- 
quired to participate in the program were assigned, by chance, either to a program group that had 
access to employment and training services and whose members were required to participate in 
the program or risk a reduction in their monthly welfare grant or to a control group that received 
no services through the program but could seek out such services from the community.^ (Control 
group members were eligible for child care assistance, similar to that offered to program group 
members, if they were participating in nonprogram activities in which they had enrolled on their 
own.) Notably, in four of the sites, there were two program groups (plus a control group). In 
three of the sites, one program group was employment-focused while the other program group 
was education-focused; in the fourth site, the two program groups varied in their case manage- 
ment staffing structure. This random assignment design assures that, within each site, there were 
no systematic differences between the background characteristics of families in the program and 
control groups when they entered the study. Thus, any subsequent differences in outcomes be- 
tween the groups — for adults, children, or families as a whole — can be attributed with confi- 
dence to the effects of the programs. These differences between outcomes are called impacts, and 
all those reported are statistically significant and hold for the whole sample unless otherwise 
noted. 



I. Overview of Findings 

No aspects of the 1 1 welfare-to-work programs studied as part of the NEWWS Evalua- 
tion were targeted directly to children. Theoretically, however, the programs could affect chil- 
dren through their effects on the parents required to participate in them, as described in the con- 
ceptual model presented below. Program effects on parents’ education levels, welfare status, em- 
ployment, earnings, and income could potentially result in changes in family routines, parents’ 
self-esteem or stress, or parents’ supervision of children or use of child care arrangements. Thus, 
as a first step, it is necessary to examine how the programs were implemented and their impacts 
on adults. 



A. Impacts on Adults 

The 1 1 programs differed in the messages that they sent to welfare recipients about how 
best to obtain and retain employment: Some stressed getting a job quickly and others stressed 



^Sample members were randomly assigned to research groups over approximately a two-year period in each 
site, starting in June 1991 in Riverside and ending in December 1994 in Portland. Thus, the results presented here 
cover the period between June 1991 (the first sample members’ entry into the study) and December 1996 (the last 
month of the two-year follow-up for the last sample members who entered the study in Portland). 
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initial investments in basic education or training. Most of the programs imposed a mandatory 
participation requirement on all recipients, and several used financial sanctions (that is, welfare 
grant reductions) extensively to enforce this mandate. All programs substantially increased par- 
ticipation in activities designed to promote employment, beyond what would have happened in 
their absence. 

Most of the programs had impacts on their targeted outcomes: Over the two-year follow- 
up period, four programs that had an education focus increased the probability that welfare re- 
cipients would obtain a high school diploma or General Educational Development (GED) certifi- 
cate. All programs decreased some aspect of welfare dependency, and 8 of the 1 1 increased two- 
year employment levels. While most programs increased individuals’ reliance on earnings, as 
opposed to welfare, net income for these individuals was largely unchanged. 

Impacts on nontargeted outcomes were found as well: Some programs led to a reduction 
in health insurance coverage, and most programs increased the use of paid child care. Across the 
programs, there were few effects on fertility or family structure, housing status, or mothers’ psy- 
chological functioning, stress, or parenting. 

B. Impacts on Children 

Impacts on children were based on data collected in three child development areas: be- 
havioral and emotional adjustment; cognitive functioning and academic achievement; and health 
and safety. In-depth data are available for preschool-age children in three of the sites (as part of a 
special Child Outcomes Study); more limited data are available for children of all ages in all 
seven sites. An examination of findings fi-om both sets of data suggests the following regarding 
all children: 

• Measured effects on children were infrequent. In addition, most effects could 
be considered small in magnitude. 

• Both favorable and unfavorable child impacts were found. Notably, however, 
they were consistently favorable in the cognitive development area, consis- 
tently unfavorable in the health area, and both favorable and unfavorable in 
the behavioral and emotional adjustment area. 

• Child impacts were not systematically different for mothers subject to em- 
ployment-focused programs than for those subject to education-focused pro- 
grams: They were not clustered in one of the two types of program, and nei- 
ther type had consistently favorable effects while the other type of program 
had consistently unfavorable effects. 

An examination of impacts on subgroups of young children (as part of the Child Out- 
comes Study) indicates the following: 

• As was true for all studied children, few child impacts were found for sub- 
groups of young children who, as of study entry, were at either high risk or 
low risk for poor development. (This analysis was conducted for preschool- 
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age children in three of the sites in which employment-focused and education- 
focused programs were operated simultaneously.) 

• The few impacts on children at higher risk for poor development were small, 
and in two of the three sites tended to be favorable for education-focused pro- 
grams and unfavorable for employment- focused programs. 

• The few impacts on children at lower risk for poor development were larger, 
tended to be unfavorable, did not tend to vary by program approach, and were 
clustered in three programs. 

Further research and longer follow-up are needed to clearly determine the mechanisms 
through which some of the programs affected children. Nonexperimental methods — which lack 
the rigor of the experimental methods that produced the findings reported above but are needed 
to examine the processes through which programs might affect children — were used in an at- 
tempt to explain the few found child impacts. The results are thus suggestive, but not definitive. 
They suggest that, for families with all school-age children, programs that place little emphasis 
on helping welfare recipients obtain good child care or that result in decreases in family income 
may tend to have unfavorable impacts on children. (There is also some indication that increases 
in employment may be connected with unfavorable child effects, but this finding held true for 
one source of data on employment and not for the other.) Most likely, these program characteris- 
tics or effects interact with each other in particular (as yet unknown) ways to affect children. 
Other examined program features or effects — whether programs were employment- or educa- 
tion-focused, the extent to which a mandatory participation requirement was enforced, increases 
in parents’ high school diploma or GED receipt, or decreases in health insurance coverage — do 
not appear, by themselves, to relate to impacts on children. Analyses of selected impacts on the 
younger children in the study also suggest that programs might affect children to the extent that 
they affect mothers’ employment and/or affect children’s home environment (for example, moth- 
ers’ psychological well-being and parenting). These analyses of preschool-age children did not 
find that increases in the use of child care, decreases in health insurance coverage, or changes in 
family income played roles in explaining the selected child impacts examined. 

Because the 11 programs operated under JOBS (Job Opportunities and Basic Skills 
Training program), which preceded TANF, they did not invoke the TANF time limit on eligibil- 
ity for welfare, try to meet its participation goals, impose full-family financial sanctions, or put in 
place the generous financial work incentives of many current programs. They also did not have 
available to them the recent and substantial increases in federal funding for child care or ex- 
panded eligibility for health insurance through Medicaid and the State Children’s Health Insur- 
ance Program. It is plausible that some of the current generation of programs will produce larger 
effects on adults than those reported here; as a result, it is possible that they may have larger ef- 
fects on children. The new policies also may result in stronger and more divergent impacts on 
children with varying initial levels of being at risk for poor development.^ 



^See Zaslow et al., 1998, and McGroder et al., 2000, for further discussion of these hypotheses. 
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The remainder of this synthesis expands on the above findings. Section II presents a con- 
ceptual model of how mandatory welfare-to-work programs might affect children. Section III 
describes aspects of child well-being examined in the NEWWS Evaluation. Section IV discusses 
the characteristics of the adults and children in the evaluation samples. Sections V, VI, and VII 
summarize program implementation and program effects on targeted and nontargeted outcomes, 
highlighting any situations where effects were different for mothers in a special Child Outcomes 
Study sample focusing on yoimg children, as compared to mothers with children of all ages. Sec- 
tion VIII — the heart of the document — presents child impacts. 



II. How Mandatory Weifare-to-Work Programs Might Affect Children 

At the outset of the NEWWS Evaluation, it was an open question as to whether, and how, 
welfare-to-work programs designed for adults would affect children. For 20 years prior to 1988, 
women receiving welfare who had children imder age 6 generally were not required to participate 
in such programs. With the passage of the Family Support Act in 1988, women with children as 
yoimg as age 3 (or as young as age 1, at state option) were newly designated as mandatory par- 
ticipants. Thus, in the early 1990s there was much interest in how welfare-to-work programs 
might affect children, particularly preschool-age children, who were seen as particularly vulner- 
able to changes in their family situation. Expectations regarding possible effects on children var- 
ied. On the one hand, given that all aspects of the programs were aimed at changing adult behav- 
ior, one might expect few effects on children, particularly if effects on adults were not dramatic. 
On the other hand, if effects on adults — for example, increased employment or participation in 
program activities such as job search or education, or increases or decreases in family income — 
were large or pervasive enough, one might expect effects on children, either positive or negative. 

The most prevalent current theories about how mandatory welfare-to-work programs 
might affect children hypothesize that program effects on adults’ employment, earnings, and in- 
come may, in turn, affect the resources available to children’s development, either positively or 
negatively .“ The resomces available to children shape the daily experiences that contribute to 
their health, safety, and development. These resomces can be material (for example, housing) or 
social (for example, interactions between mothers and children).* As a positive example, welfare- 
to-work programs that raise income might allow families to afford better and safer housing. Ad- 
ditionally, employment may improve mothers’ self-esteem, enhancing their ability to be a role 
model for their children. As a negative example, the reduction of working mothers’ time at home 
may result in decreased overall supervision of their children. Additionally, the requirement to 
participate in a welfare-to-work program or the experience of holding a new job may result in 
increased stress for mothers, affecting parenting practices. 



“See, for example, Wilson, Ellwood, and Brooks-Gunn, 1995; Zaslow et al., 1995; Zaslow et al„ 1998; and 
McGroder et al., 2000. The conceptual model used here was developed by Child Trends early in the NEWWS 
Evaluation, and has undergone refinements over subsequent years. 

^See Haveman and Wolfe, 1995. 
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Figure 1 depicts the theoretical model described above.® The pivotal box in this model, 
labeled “Targeted Outcomes” (box C), represents the adult outcomes targeted by welfare and 
employment policies and programs (box A) that can be affected through implementation of the 
policies and programs (box B). Changes in the targeted outcomes, which can affect the resources 
available to families as well as family socialization patterns, can produce effects on nontargeted 
outcomes (box D). These nontargeted outcomes represent other avenues through which child 
outcomes (box E) might be affected.’ 

As evident in the model, it is important to establish that impacts on the targeted outcomes 
or nontargeted outcomes exist. If impacts on children are found, but no impacts are apparent for 
the targeted or nontargeted outcomes, it will be unclear what led to the child impacts. (While the 
rigorous NEWWS research design can provide solid evidence about the existence of impacts on 
children, it does not allow firm causal inferences to be made about the processes through which 
mandatory welfare-to-work programs may affect children’s well-being.) Associations found 
among program features, adult impacts, environmental effects, and child impacts, however, can 
give clues about possible pathways of effects. These can then be investigated through further re- 
search and result in modifications to the conceptual model. 



III. Aspects of Child Well-Being Examined in the NEWWS Evaluation 

In order to allow sufficient time for the full effects of the NEWWS Evaluation programs 
to manifest themselves, sample members in the evaluation are being followed for five years fi'om 
the time they entered the study. Comprehensive data on economic outcomes, including informa- 
tion on quarterly unemployment insurance-reported earnings and monthly welfare and Food 
Stamp payments, are being collected over the five-year follow-up period for 44,569 single par- 
ents (the full sample) who have been randomly assigned to research groups across the seven 
evaluation sites. (See box A in Figure 2.) At this point in the evaluation, economic outcome data 
covering the first two years of follow-up have been analyzed and results have been made public. 

In addition, a wealth of information was collected through interviews with a subsample of 
9,675 individuals, randomly chosen fi'om all research groups in all seven sites, two years after 
they entered the study. Interviewees in this client survey sample were asked about their experi- 
ences in the program, educational attainment, family composition, housing status, employment 
and wage progression, and total family income, as well as about their children’s behavior, school 
progress, and health and safety.® (See box B in Figure 2.) 



*The primary sources for the pathways in this model are correlational studies of the relationships between in- 
come, employment, child care, and child outcomes; previous work by MDRC and others on interventions and their 
effects on income and employment; and the underlying theories about how welfare-to-work or employment policies 
might affect children. The model depicts the effects of parent actions on children, but not the effects of children on 
parents. This is likely an overly linear representation of the relationships between parents and children. 

’The nontargeted outcomes in box D include those that theoretically could affect child outcomes. Measmement 
of some of these nontargeted outcomes is difficult, if not impossible; as a result, not all of them were examined in 
the NEWWS Evaluation. 

*A subsample of these interviewees also took adult reading and math achievement tests. 
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Figure 1 

Conceptual Model of the Pathways through Which Welfare and Employment Policies and Programs Influence Child Outcomes 
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National Evaluation of Welfare-to-Work Strategies 
Figure 2 

Sample Sizes and Data Sources for 
Child Outcome Analyses in the NEWWS Evaluation 



0 - 



Full Impact Sample 

All 7 Sites 
N = 44,569 

■ Administrative records-based economic outcome measures 
- No child-specific outcome measures 



(®> 



Two-Year Client Survey Sample 
All 7 sites 
N = 9,675 

- Parent report of economic and other outcome measures 
- Parent report of children's behavior, school progress, and health and safety 



0 



Child Outcomes Study (COS) Sample 
(subset of mothers with children aged 3-5 at baseline) 

3 sites: Atlanta, Grand Rapids, and Riverside 

N = 3,018 

Expanded COS measures for "focal" child: 

- Battery of cognitive tests given to child 

- Detailed maternal report of child's behavioral status and simple rating of 
child's physical health 

- Detailed maternal report of mother's well-being and family situation 
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For a subset of the surveyed individuals (3,018 surveyed sample members in three of the 
seven evaluation sites who had a child aged 3 to 5 at study entry), additional, more detailed data 
were collected at the time of the interview for the young child.® (If a family had more than one 
child aged 3 to 5, one was randomly selected to be studied in depth.) (See box C in Figure 2.) 

These “focal” children, who would have been between about ages 5 and 7 at the two-year 
interview, were given a direct assessment of cognitive development, and the mothers completed 
multiple and more detailed measures regarding the children’s behavioral and emotional adjust- 
ment and physical health and safety. Mothers in the Child Outcomes Study (COS) sample were 
also asked additional questions concerning nontargeted aspects of the welfare-to-work programs: 
for example, mothers’ psychological well-being, child care arrangements, fathers’ involvement 
with children and the payment of child support, and the family’s home and neighborhood envi- 
ronment. 

Data relating to all children of the 9,675 mothers interviewed in seven sites and to the 
young focal children of the 3,018 mothers interviewed in three sites were collected for three child 
developmental domains or outcome areas: behavioral and emotional adjustment; cogmtive func- 
tioning and academic achievement; and health and safety. Appendix A details the measures used 
at the two-year follow-up point to assess changes in each of these areas. Data collection on chil- 
dren (and adults) is ongoing in the NEWWS Evaluation,'® and Box 1 describes the future analy- 
ses planned. 



IV. Characteristics of Adults and Children in the Samples 

In most aspects, the samples in the seven NEWWS Evaluation sites are diverse. Across 
all sites, almost all adult sample members were female single parents and, on average, 30 years 
old with two children at the time of study entry. The samples in Grand Rapids, Detroit, and 
Oklahoma City included teen parents, who represented, at most, 10 percent of each site’s full 
sample. In Grand Rapids, Detroit, Oklahoma City, and Portland, mothers had children as young 
as age 1 at study entry; in these four sites, about two-fifths of the sample members entered the 
program when their youngest child was under age 3. The remainder of the samples in these four 
sites, as well as the full samples in the other three sites, were about evenly split between parents 
whose youngest child, as of study entry, was aged 3 to 5 and those whose youngest child was 
aged 6 or over. Depending on the site, between one-third and one-half of sample members gave 
birth to their first child when they were teenagers." 



^These additional data were actually collected for 3,194 surveyed individuals, but 176 of the families were 
deemed inappropriate, for various reasons, for the in-depth Child Outcomes Study, resulting in a sample size of 
3,018. 

*®Additional data already collected include in-depth information on the implementation of the programs — the 
extent of individuals’ participation in various program activities, the quality of the activities, and staff practices and 
opinions — as well as on the costs of different aspects of welfare-to-work programs. 

**Note that parents with a severely ill or disabled child were generally not mandated to participate in welfare-to- 
work programs in the early to mid 1990s. While the proportion of families exempted from the participation re- 

(continued) 
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Box 1: Future Analyses of Children in the NEWWS Evaluation 
Will Cover Five Years of Follow-Up 
and Include Elementary School Teacher Assessments 



While extensive and rich data on both adult and child development outcomes were col- 
lected in the NEWWS Evaluation at the two-year follow-up mark, two years is not enough time 
to fully assess program impacts on child (or adult) outcomes. Another round of data collection 
is in progress for study sample members at their five-year follow-up point. These data will in- 
dicate whether the impacts on children observed at the two-year point persist, grow, or decline 
by the end of five years. In addition, new program impacts on child outcomes may emerge. 

At the five-year follow-up point, a small set of questions will again be asked of parents 
with children of all ages. In addition, for the sample of preschool-age focal children in three 
sites (who will be 8 to 10 years old at this point), math and reading skills will be assessed, and 
elementary school teachers will report on scholastic performance (whether performing at grade 
level, whether skipped or repeated a grade, and how performing in comparison to classmates). 
Finally, parents and teachers — and the focal children themselves — will report on focal chil- 
dren’s behavior, maturity, social competence, and engagement in school. 



The ethnic make-up of the samples varied across the sites, reflecting the ethnic composi- 
tion of the localities fi-om which the samples were drawn. In Atlanta and Detroit, almost all sam- 
ple members were Afi-ican-American. About half of the sample members in Grand Rapids, Riv- 
erside, Columbus, and Oklahoma City, and two-thirds of those in Portland, were white. Only 
Riverside had a substantial portion (one-third) of Hispanics. 

Slightly more than one-half of sample members had a high school diploma or General 
Educational Development (GED) certificate when they entered the program, and in all seven 
sites at least some study enrollees had some college or post-secondary schooling. On average, 
however, sample members had completed just 1 1 years of school as of study entry. None of the 
programs served populations who, as a whole, had much work history; fewer than half the indi- 
viduals in all sites but Oklahoma City had worked at some point during the year prior to study 
entry. In all sites except Oklahoma City, between a quarter and a half of sample members had 
received welfare cumulatively for at least five years.'^ Furthermore, up to a quarter of sample 
members in any site met a definition of “most disadvantaged”; that is, they did not have a high 
school diploma or GED, lacked any work history in the year prior to enrolling in the program, 
and already had received welfare cumulatively for two years or more before entering the study. 



quirement for this reason during this time period was very small, such families would not have been included in the 
samples examined in the NEWWS Evaluation. 

'^The Oklahoma City sample included only welfare applicants (and not recipients), who were thus more likely 
to have more prior work experience and less prior welfare receipt. 
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In contrast to the full samples of parents described above, adults in the Child Outcomes 
Study (COS) sample (a subset of parents in the Atlanta, Grand Rapids, and Riverside samples) 
consisted solely of mothers with preschool-age children. At study entry, the mothers’ average 
age was 27 in the Grand Rapids sample and 29 in the Atlanta and Riverside samples, below the 
average age of parents in the full sample.'^ Nevertheless, COS mothers had, on average, a 
slightly higher number of children than parents in the full sample. In addition, these mothers 
were less likely than those in the full sample to have ever been married and more likely to have a 
high school diploma or GED. Finally, in two of the three sites (Atlanta and Riverside), COS 
mothers were less likely than those in the full sample to have ever worked full time for six 
months or more for one employer. Some of these sample differences — for example, the younger 
average age of COS mothers and their relative lack of work experience — are typical of mothers 
with young children. Other sample differences — for example, COS mothers being more likely 
to have a high school diploma or GED — reflect the fact that women on welfare with all older 
children are often those who have more barriers to finding a job and leaving welfare, while 
women with young children, particularly those not married, are perhaps more likely to have just 
recently started receiving welfare, and the most advantaged of this group will leave welfare 
within a few years. 

The children of adult sample members across the seven NEWWS Evaluation sites ranged 
in age from 1 to 17. Approximately 49 percent of all families had all school-age children, 23 per- 
cent had no school-age children, and 28 percent had both school-age and preschool-age children. 
As noted above, about two-fiflhs of the families in four sites included a child as young as age 1 
or 2. At the other end of the age spectrum, approximately one-third of the families in all seven 
sites included at least one high school-age child, that is, a child between ages 13 and 17. Other 
characteristic data for children are available only in Atlanta and only for a small number of 
young children who were between ages 3 and 5 at study entry. (See Box 2.) 



V. Program Implementation 

As shown in Figure 1, features of program implementation lead directly to the targeted 
outcomes of welfare and employment policies and programs and can also influence nontargeted 
outcomes. This section discusses the program services, mandates, and activities shown in box B 
of the figme. Results for the full sample are highlighted. Where evident, distinctions are drawn 
between program implementation as it was experienced by the seven-site client survey sample 
(for which impacts on primarily school-age children are available) and by the COS sample in 
three of the evaluation sites (for which impacts on preschool-age children are available). 

• The evaluation sites implemented very different welfare-to-work pro- 
grams; in fact, in four of the sites, two types of programs were operated 
within each site, to allow the study to rigorously compare the effects of 
specific program approaches. 



'^Tests of statistical significance were not conducted on the differences in sample characteristics discussed in 
this paragraph. 
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Box 2: In At Least One Aspect of Development, Preschool-Age Children 
in the Atlanta COS Sample at Study Entry 
Were More Disadvantaged Than Children in National Samples 



In the Atlanta NEWWS Evaluation site, a special descriptive study was conducted 
close to the start of the evaluation, in order to describe the lives and circumstances of a sample 
of welfare families with preschool-age children and to inform policymakers about the devel- 
opmental status of young children receiving welfare. (See Moore et al., 1995.) Approximately 
three months after sample mothers entered the study, interviews were conducted in the home 
and direct assessments were made of their children’s cognitive development in the areas of re- 
ceptive vocabulary and school readiness. In addition, mothers reported on their children’s so- 
cioemotional development and health status. A brief summary of how this sample of 790 chil- 
dren were faring at roughly the start of the NEWWS Evaluation follows. 

Cognitive development: The Peabody Picture Vocabulary Test-Revised was used to assess 
cognitive development. This measure is highly correlated with measures of both intelligence 
and school achievement and is a predictor of IQ. Mean scores of children in the Atlanta sample 
on this measure were lower (by .4 of a standard deviation) than the mean scores of African- 
American children from welfare families in a national sample. (Comparisons were made solely 
for African-American children because of the possibility of racial bias with this measure.) 
Children in the Atlanta sample scored approximately two-thirds of a standard deviation below 
nonpoor children in a national sample. 

Socioemotional development: Using the Personal Maturity Scale, mothers described their 
children as showing fairly high levels of maturity. The average score on this scale was 8 out of 
a possible 10, with 10 indicating the highest level of maturity. National results using a compa- 
rable scale were not available. 

Child health: Approximately 50 percent of the children were described by their mothers as in 
excellent health with no limiting conditions. This portrayal of children’s health is in keeping 
with the fact that serious child health problems could, at the time, result in mothers being ex- 
empted from welfare-to-work programs. Using a similar composite rating from a national sam- 
ple, 38 percent of children in families on welfare, 42 percent of those in families who were 
poor but not on welfare, and 52 percent of those in nonpoor families were described as in ex- 
cellent health with no limiting conditions. 



For analysis purposes, distinctions are drawn in the evaluation between employment- 
focused programs and basic education-focused programs, as well as between programs with high 
and low levels of enforcement of the participation mandate. Taking into account these two di- 
mensions of program characteristics, as well as the types of program activities to which welfare 
recipients were initially assigned, four categories of welfare-to-work program approaches 
emerge, shown in Table 1. 
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National Evaluation of Welfare-to-Work Strategies 

Table 1 



Categorizing NEWWS Programs, by Approach, 



First Activity, and Enforcement Level 


Employment-Focused Approach 


Education-Focused Approach 


Job Search First: 
High Enforcement 


Varied First Activity: 
High Enforcement 


Education or 
Training First: 
High Enforcement 


Education or 
Training First: 
Low Enforcement 


Atlanta LFA 
Grand Rapids LFA 
Riverside LFA 


Portland 


Atlanta HCD 
Grand Rapids HCD 
Riverside HCD 
Columbus Integrated 
ColumbusTraditional 


Detroit 

Oklahoma City 



The distinction between employment- and education-focused approaches is central to the 
NEWWS Evaluation. To promote ongoing work and self-sufficiency among welfare recipients, 
states have traditionally implemented one or the other of these two approaches. The employment- 
focused approach emphasizes placing people in jobs quickly, even at low wages, reflecting a 
view that the workplace is where welfare recipients can best build their work habits and skills. 
The education-focused approach, which emphasizes education and training as a precursor to em- 
ployment, is based on the belief that the required skill levels for many jobs are rising and that an 
investment in the “human capital” of welfare recipients will allow them to obtain better and more 
secure jobs. The two approaches convey different messages to welfare recipients about the best 
route to self-sufficiency, and they emphasize different program components. One aim of the 
NEWWS Evaluation is to determine the relative effects of the two approaches on both adults and 
their children. 

Four of the sites in the evaluation — Atlanta, Grand Rapids, Riverside, and Columbus — 
operated two different programs simultaneously, to enable rigorous side-by-side tests of the 
comparative effectiveness of various approaches. Each of the first three of these sites imple- 
mented a “labor force attachment” (LFA) program as well as a “human capital development” 
(HCD) program, versions of employment-focused and education-focused programs that magni- 
fied the differences between the two types of approaches. The fourth site, Columbus, imple- 
mented a program using a “traditional” case management model, in which welfare eligibility and 
employment program fimctions were performed by separate sets of staff, as well as a program 
using an “integrated” case management model, in which these two functions were performed by 
the same staff. In the remaining three sites in the evaluation — Oklahoma City, Detroit, and Port- 
land — the sites’ established programs were studied. In all, 1 1 programs were examined in the 
seven NEWWS Evaluation sites. 





- 13 - 



• Employment-focused programs differed significantly from education- 
focused programs in the message that was sent to welfare recipients about 
how to obtain employment and in the sequence and emphasis of required 
program activities. These differences were evident in all three studied 
samples: the full sample, the client survey sample, and the Child Out- 
comes Study (COS) sample. 

The four employment- focused programs (see Table 1) provided job search assistance to a 
large segment of their caseload and encouraged enrollees to find work as quickly as possible. The 
three LFA programs, however, differed from the Portland program in two important ways. First, 
the LFA programs routinely assigned individuals to job search as their first activity, whereas 
Portland offered GED preparation classes to people who were thought to have a good chance of 
attaining a GED certificate relatively quickly. Second, Portland case managers, more often than 
those in the LFA programs, encouraged enrollees to hold out for a job that paid well above the 
minimum wage and offered the best chance for long-lasting and stable employment. In contrast, 
LFA case managers, especially in Riverside, stressed the value of taking any job, even a low- 
paying one, and trying to advance. 

In the education- focused programs, a large percentage of program enrollees were initially 
assigned to some type of skill-building activity. Their first assignments depended, in part, on 
their educational levels on entering the program. Those with low reading or math skills were as- 
signed to adult basic skills classes; those with higher skills but lacking a high school diploma or 
GED were assigned to GED preparation classes; and non-English speakers could be assigned to 
English as a Second Language (ESL) classes. Finally, those with a high school diploma or GED 
could be assigned to vocational training or employment-oriented skills courses at local commu- 
nity colleges. All in all, however, assignments to GED preparation or basic education courses 
predominated, vocational training program assignments were less common, and enrollment in 
college was minimal. Riverside’s HCD program was unique among this group in that it did not 
serve high school graduates and GED holders who, at program entry, scored above minimum 
levels in reading and math tests. 

• The 11 programs varied widely in the degree to which a participation 
mandate was enforced and in their use of financial sanctions (welfare 
grant reductions), but the six programs in which COS sample members 
participated enforced the mandate and had moderate to high sanction 
rates. 

As specified in the research design, no control group members were subject to a participa- 
tion requirement and, as a result, none of them experienced any sanctions. In contrast, a wide 
cross section of program group members were enrolled in most programs, and participation was 
monitored closely. Failure to participate could result in a sanction, that is, a reduction in a fam- 
ily’s total welfare grant. Sanction rates were high in four programs (Grand Rapids LFA and HCD 
and Columbus Integrated and Traditional), where at least 26 percent of sample members were 
sanctioned at some point during the two-year follow-up period, and low in two programs (Detroit 
and Oklahoma City), where less than 5 percent of sample members were ever sanctioned. Sanc- 
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tion rates for the remaining programs fell between these two extremes and were considered to be 
moderate to somewhat high. 

• All 11 programs increased participation levels in activities designed to 
promote employment during the two-year follow-up period. 

Many control group members took part in employment-related activities, such as basic 
education, skills training, post-secondary education, or formal job search, on their own initiative 
at some point during the two-year follow-up period. All programs, however, were able to in- 
crease participation levels in such activities above those achieved by the control groups. Of the 
nine programs with at least a moderate enforcement of the participation mandate, all but one 
(Grand Rapids LFA) produced large impacts on participation, ranging from 21 percentage points 
(Grand Rapids HCD and Columbus Traditional) to 40 percentage points (Riverside HCD).“‘ As 
expected, all of the employment-focused programs produced large increases in participation in 
job search activities, but two also produced small increases in participation in education and one 
produced a small increase in training. Most of the education-focused programs raised participa- 
tion levels in education or training. These programs also, to a lesser extent, increased participa- 
tion in job search.'* 



VI. Impacts on Targeted Outcomes 

Figure 1 suggests that numerous features of mandatory welfare-to-work programs are hy- 
pothesized to directly impact the adults subject to such programs and that these targeted effects 
on adults can potentially, through effects on nontargeted outcomes, influence child well-being. 
This section briefly summarizes effects on targeted outcomes for the 1 1 programs in the seven 
evaluation sites. The effects of die programs are measured by comparing outcomes for program 
groups with those of control groups; the resulting differences, or impacts, can be confidently 



'^In this section, as well as the following section on targeted outcomes, the benchmarks for characterizing the 
magnitude of program impacts are based on ranges of impact findings from previous experimental evaluations of 
welfare-to-work programs. The specific thresholds vary by outcome. Impacts on participation of 20 percentage 
points or more are considered “large,” impacts of 10 to 20 points are “moderate,” and impacts below 10 points are 
“small.” For measures of high school diploma or GED attainment, welfare expenditures, and employment, impacts 
of 10 percentage points or more are considered “large,” impacts of 5 to 10 points are “moderate, and impacts be- 
low 5 points are “small.” Impacts on earnings or income of more than $900 per year (or $1,800 over two years) are 
considered “large,” impacts of $300 to $900 per year (or double these amounts over two years) are “moderate,” and 
impacts of less than $300 per year are “small.” 

’’Length of stay in the programs, and thus participation in program activities, was shorter than in voluntary pro- 
grams, but not shorter than in typical mandatory welfare-to-work programs. This reflects the fact that people cycle 
off welfare frequently. In the studied programs, most people did not remain on welfare continuously, and thus most 
were not subject to the program participation requirements, for the full two years of follow-up. In the seven pro- 
grams for which detailed participation statistics have been published so far, the length of time during the two-year 
follow-up period that program group members spent enrolled in the programs ranged, on average, from 1 1 months 
in the Riverside LFA and HCD programs to 17 months in the Atlanta HCD program. The average number of 
months of participation in program activities for only those who ever participated at all ranged from 3 months in the 
Riverside LFA program to 9 months in the Atlanta HCD program. (See Hamilton et al., 1997, and Scrivener et al., 
1998.) 
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viewed as the effects of the programs.'® As in the previous section, while most shown results are 
for the full sample, contrasts between impacts for the client survey sample and for the COS sam- 
ple, where they exist, are highlighted.” 

• Four of the 11 programs increased the probability that sample members 
would obtain a high school diploma or GED, including two of the three 
education-focused programs to which COS sample mothers were subject 

While most education- focused programs increased participation in basic education among 
those who entered the study without a high-school diploma or GED, only three of these programs 
(Grand Rapids and Riverside HCD and Columbus Traditional) increased GED certificate attain- 
ment. For all sample members, not just those who entered the study without these education cre- 
dentials, impacts on GED receipt in the three programs ranged from 3 to 8 percentage points. 
(Table 2 shows these impacts for the several different samples for which child impacts are dis- 
cussed later in this document.) Portland’s employment-focused program, which used a varied 
first activity approach, achieved similar gains in GED receipt. (The other three employment- 
focused programs had no effect on GED attainment.) 

• All 11 programs reduced some aspect of welfare dependency to some de- 
gree, but among COS sample mothers welfare reductions were not as 
universal. 

For the full sample, seven programs decreased cumulative welfare expenditures by 10 
percent or more relative to the control groups, a historically large effect. (See Appendix B.) The 
Portland and Grand Rapids LFA programs produced imusually large decreases of 17 and 19 per- 
cent, respectively. The Riverside LFA welfare expenditure impact was also large (14 percent). 
Among COS sample mothers, four of the six programs to which they were subject reduced wel- 
fare dependency. For this sample, the Grand Rapids and Riverside LFA programs decreased cu- 
mulative welfare expenditures by at least 15 percent, a large impact; the Atlanta LFA and River- 
side HCD programs decreased expenditures by a smaller percentage. Finally, while welfare ex- 



'*This section focuses on economic impacts as measured by the administrative records databases described in 
Section n. McGroder et al., 2000, Chapter 9, in presenting economic impacts for the adults in the Child Outcomes 
Study, primarily used client survey data. Each of these data soiuces has its advantages and disadvantages. Measiues 
of income using admiiustrative records data, for example, cover the entire two-year follow-up period, but include 
income only from cash welfare. Food Stamps, and Unemployment Insurance-reported earnings. Measiues of income 
using client survey data include many more potential soiuces of income, but cover only one month (the last month 
of the two-year follow-up period). As a result of these differences, some summary statements about economic im- 
pacts in McGroder et al., 2000, differ somewhat from those in this document. 

’’Impacts could differ for the samples for several reasons. First, the client survey sample includes individuals 
from all seven evaluation sites, while the COS sanq>le was drawn from only three of the sites. Second, even within 
the three sites in which the COS is nested, there are demographic differences between sample members with pre- 
school-age children and those with older or even yoimger children, as discussed in Section IV. Finally, in general, 
different data soiuces and response rates can produce different impacts for the client survey sample compared with 
the full sample. See Appendix E in Freedman et al., 2000, for a discussion of the reliability and generalizability of 
results based on the client survey and Appendix F for a comparison of impacts estimated from survey and imem- 
ployment insiuance data. 
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National Evaluation of Welfare-to-Work Strategies 
Table 2 

Impacts on Selected Targeted Outcomes, by Sample 



Site 


Full Sample 


Client Survey Sample: 
Families with All 

Client Survey Sample School-Age Children 


COS Sample 


Control 

Group 


Impact 


Control 

Group 


Impact 


Control 

Group 


Impact 


Control 

Group 


Impact 








Received a high school diploma or GED f%) 






Atlanta LFA 


N/a‘ 




1.2 


0.9 


1.2 


1.3 


. 




Grand Rapids LFA 






4.2 


-1.8 


2.7 


-2.2 


5.1 


-2.8 * 


Riverside LFA 






2.4 


-0.9 


1.9 


-1.2 


- 




Portland 






1.8 


4.3 ** 


0.4 


1.6 






Atlanta HCD 






- 1.2 


1.0 


1.2 


0.5 


_ 




Grand Rapids HCD 






4.2 


2.5 •* 


2.7 


1.4 


5.1 


3.6 •• 


Riverside HCD 






2.4 


8.3 *** 


1.9 


6.1 *** 


2.7 


10.3 


Columbus Integrated 






2.9 


2.1 


2.0 


1.3 






Columbus Traditional 






2.9 


3.3 •• 


2.0 


3.4 * 






Detroit 






5.6 


1.5 


3.6 


-2.3 






Oklahoma City 






4.3 


3.4 


2.3 


0.8 












Ever employed in years 1 or 


im 






Atlanta LFA 


61.6 


4.5 ••• 


64.3 


2.3 


63.8 


1.5 


65.4 


4.2 


Grand Rapids LFA 


70.1 


7.6 ••• 


72.0 


8.3 *** 


70.3 


4.1 


75.1 


11.0 *** 


Riverside LFA 


45.0 


15.1 ••• 


46.0 


18.2 *** 


53.4 


10.6 ••• 


37.9 


24.7 ••• 


Portland 


60.9 


11.2 ••• 


63.6 


6.7 * 


67.5 


5.5 






Atlanta HCD 


61.6 


2.8 •• 


64.3 


4.2 ** 


63.8 


2.5 


65.4 


6.9 •• 


Grand Rapids HCD 


70.1 


5.3 ••• 


72.0 


7.5 *** 


70.3 


7.8 •• 


75.1 


6.1 • 


Riverside HCD 


38.9 


9.3 ••• 


38.5 


11.2 *** 


45.1 


8.2 •• 


35.8 


16.1 ••• 


Columbus Integrated 


72.2 


1.7 


70.2 


1.1 


69.9 


-0.3 






Columbus Traditional 


72.2 


1.3 


70.2 


3.1 


69.9 


5.1 






Detroit 


58.2 


4.1 


53.5 


10.1 *• 


48.5 


7.0 






Oklahoma City 


65.0 


-0.9 


69.3 


-2.4 


69.5 


-4.6 












Average total earnings in years 1 and 2 (S) 






Atlanta LFA 


5,006 


813 •** 


5,412 


548 


5,482 


490 


5,385 


744 


Grand Rapids LFA 


4,639 


1,035 ••• 


6,136 


566 


7,234 


-404 


5,568 


1,487 •• 


Riverside LFA 


4,213 


1,276 ••• 


3,962 


1,654 ••• 


4,837 


778 


3,098 


2,199 ••• 


Portland 


5,291 


1,842 ••• 


5,170 


1,317 • 


5,094 


2,592 •• 






Atlanta HCD 


5,006 


496 •• 


5,412 


301 


5,482 


273 


5,385 


382 


Grand Rapids HCD 


4,639 


580 •• 


6,136 


15 


7,234 


-556 


5,568 


372 


Riverside HCD 


3,133 


317 


2,995 


729 • 


3,443 


700 


2,860 


949 • 


Columbus Integrated 


6,892 


673 •• 


6,984 


539 


7,807 


-124 






Columbus Traditional 


6,892 


677 ••• 


6,984 


456 


7,807 


618 






Detroit 


4,001 


367 • 


3,547 


359 


3,132 


418 






Oklahoma City 


3,514 


5 


3,920 


523 


4,775 


626 








O 

40 
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Table 2 (continued) 



Site 


Full Sample 


Client Survey Sample: 
Families with All 

Client Survey Sample School-Age Children 


COS Sample 


Control 

Group 


Impact 


Control 

Group 


Impact 


Control 

Group 


Control 
Impact Group 


Impact 








Combined income in year 2 (SI** 






Atlanta LFA 


7,549 


191 


7,867 


347 * 


7,561 


406 


8,396 


263 


Grand Rapids LFA 


7,746 


-303 ** 


8,468 


-511 * 


8,492 


-1,232 *** 


8,810 


-230 


Riverside LFA 


7,874 


-358 *** 


8,301 


-382 


8,173 


-1,129 *** 


8,857 


102 


Portland 


8,110 


238 


8,316 


-242 


7,003 


1,396 * 






Atlanta HCD 


7,549 


235 


7,867 


253 


7,561 


177 


8,396 


367 


Grand Rapids HCD 


7,746 


-91 


8,468 


-225 


8,492 


-616 


8,810 


-39 


Riverside HCD 


7,768 


-619 *** 


8,349 


13 


8,053 


-410 


8,913 


169 


Columbus Integrated 


8,332 


-41 


8,596 


-97 


8,716 


-243 






Columbus Traditional 


8,332 


29 


8,596 


129 


8,716 


110 






Detroit 


8,892 


101 


8,541 


275 


7,597 


-11 






Oklahoma City 


5,238 


-137 


6,055 


30 


6,346 


153 












Income at or above the poverty level 


in year 2 1%!“ 






Atlanta LFA 


12.9 


1.6 


13.9 


1.9 


13.6 


2.4 


14.5 


1.5 


Grand Rapids LFA 


13.5 


1.2 


16.9 


-0.9 


17.0 


-3.3 


18.5 


0.0 


Riverside LFA 


16.5 


1.0 


17.4 


1.1 


19.3 


-3.0 


15.4 


6.3 ** 


Portland 


16.6 


4.0 *** 


18.2 


2.3 


14.8 


13.2 ** 






Atlanta HCD 


12.9 


2.0 * 


13.9 


1.2 


13.6 


0.0 


14.5 


3.2 


Grand Rapids HCD 


13.5 


0.3 


16.9 


-0.6 


17.0 


-3.5 


18.5 


-2.2 


Riverside HCD 


13.6 


0.2 


13.6 


1.6 


15.6 


-0.2 


13.6 


3.6 


Columbus Integrated 


20.7 


0.0 


21.3 


1.6 


25.3 


-2.3 






Columbus Traditional 


20.7 


0.3 


21.3 


-1.3 


25.3 


-0.8 






Detroit 


15.9 


1.2 


15.7 


1.5 


16.4 


-4.4 






Oklahoma City 


7.2 


0.5 


8.1 


1.7 


12.7 


-3.1 










Income below 50% of the poverty level in year 2 (%) 


b 




Atlanta LFA 


31.8 


1.4 


28.7 


0.1 


30.8 


2.1 


24.5 


-2.6 


Grand Rapids LFA 


26.3 


5.0 *** 


26.9 


5.8 ** 


31.1 


9.6 ** 


25.2 


3.7 


Riverside LFA 


33.3 


5.3 *** 


27.4 


7.1 *** 


28.5 


11.3 *** 


22.1 


4.7 


Portland 


31.2 


2.1 


29.4 


6.3 


37.0 


-1.7 






Atlanta HCD 


31.8 


1.6 


28.7 


2.0 


30.8 


6.1 ** 


24.5 


-3.3 


Grand Rapids HCD 


26.3 


3.4 ** 


26.9 


1.4 


31.1 


4.0 


25.2 


-4.6 


Riverside HCD 


33.2 


6.3 *** 


26.0 


2.6 


27.5 


7.7 ** 


22.5 


1.0 


Columbus Integrated 


31.0 


2.6 ** 


30.1 


0.5 


30.7 


2.0 






Columbus Traditional 


31.0 


1.8 


30.1 


-1.3 


30.7 


1.2 






Detroit 


19.5 


-0.1 


23.0 


-6.2 * 


29.2 


-3.6 






Oklahoma City 


49.9 


2.1 ** 


42.6 


3.7 


45.2 


4.8 







(continued) 
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Table 2 (continued) 

SOURCES: MDRC and Child Trends calculations from unemployment insurance (UI) earnings records, welfare records 
and the Two-Year Client Survey. 

NOTES: Dashes indicate that sample sizes were too small to generate a reliable impact estimate. 

“This survey-based measure was not available for the full sample. 

**This measure of income does not include an estimate of the earned income tax credit (EITC). 

Riverside limited enrollment in its HCD program to individuals determined by program regulations to need basic 
education, because they lacked a high school diploma or GED certificate, attained low scores on a reading or math test 
administered at program entry, or had limited proficiency in English. As a result, control group means differ for the 
Riverside LFA and HCD programs. 

A two-tailed t-test was applied to differences between outcomes for the program and control groups. 

Statistical significance levels are indicated as = 10 percent, = 5 percent, and *** = 1 percent. 



penditure reductions were found for the full samples in the Atlanta and Grand Rapids HCD pro- 
grams, they were not evident for the COS samples in these same programs. 

• Eight of the 11 programs increased two-year employment levels. Among 
COS sample members, five of the six programs to which they were sub- 
ject increased employment, and impacts on employment rates were gen- 
erally substantially higher for this sample than for the client survey sam- 
ples in these five programs. 

A majority of control group members worked for pay at some point during the two-year 
follow-up period. For the client survey sample, three of the employment-focused programs (all 
except Atlanta LFA) and four of the seven education-focused programs (Atlanta, Grand Rapids, 
and Riverside HCD; and Detroit) increased two-year employment levels. (See Table 2.) Among 
COS sample members, all programs except the Atlanta LFA program increased two-year em- 
ployment levels. Employment impacts ranged from 6 percentage points in the Grand Rapids 
HCD program to 25 percentage points in the Riverside LFA program. These employment im- 
pacts were substantially larger than those for the client survey sample in all but one of the five 
programs. The biggest difference occurred in the Riverside LFA program, where the impact for 
the COS sample was nearly 7 percentage points higher than the impact for the client survey sam- 
ple. 



• Nine of the 11 programs produced two-year earnings gains, although 
many of these impacts were not statistically significant for the client sur- 
vey sample. Impacts on total two-year earnings were found for the COS 
sample in three of the six programs: Riverside LFA and HCD and Grand 
Rapids LFA. 

For the client survey sample, earnings gains for three of the programs (Riverside LFA 
and HCD and Portland) were moderate and statistically significant, while earnings gains for the 
other programs were smaller and not statistically significant. (All but two of the 1 1 programs 
produced two-year earnings gains for the much larger full samples.) (See Table 2.) For COS 
sample mothers, impacts on total two-year earnings were found in the Riverside LFA and HCD 
and Grand Rapids LFA programs. The largest earnings impact occurred in the Riverside LFA 
program where, on average, mothers in the COS program group earned an impressive 71 percent 
more than their control group counterparts. 

• While most programs increased sample members’ reliance on earnings, 
as opposed to welfare, their net incomes were largely unchanged. As a re- 
sult, the programs lifted few additional families above the poverty line. 

This was the case for the client survey sample as well as the COS sample. 

Across all 1 1 programs in the second year of follow-up, control group members in the 
client survey sample averaged between $6,055 (Oklahoma City) and $8,596 (Columbus) in com- 
bined income from earnings, welfare, and Food Stamps. Few programs substantially altered these 
combined income levels; in general, reductions in welfare. Food Stamps, and other benefits 
matched or exceeded earnings gains. (See Table 2.) Including estimates of the earned income tax 
credit (EITC) as income (not shown in Table 2) produced little change in this finding. Interest- 
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ingly, one program (Riverside LFA) increased the proportion of COS sample members with in- 
comes at or above the poverty level by 6 percentage points, a result not found for the Riverside 
LFA full sample. For COS sample members, there were no program effects on child support 
awards or payment. 

• For the client survey sample, two programs had the effect of pushing a 
proportion of families deeper into poverty. For families in the COS sam- 
ple in these two programs, this result was not statistically significant. 

The Grand Rapids and Riverside LFA programs increased, by 6 and 7 percentage points, 
respectively, the proportion of sample members living deeply in poverty, that is, below 50 per- 
cent of the poverty line. (See Table 2.) While increases on this measure are apparent for the COS 
samples in these two programs, they are not statistically significant. 



VII. Impacts on Nontargeted Outcomes 

As shown in Figure 1 , mandatory welfare-to-work programs can affect nontargeted out- 
comes (box D) through effects on the targeted outcomes (box C). For example, an increase in 
employment, if the increase is in jobs that do not provide health benefits, could affect the propor- 
tion of people with health care coverage. Program implementation features (box B) can also af- 
fect nontargeted outcomes (box D). For example, interactions between welfare recipients and 
case managers that stress the importance of quality child care, or networks among welfare recipi- 
ents that develop through job search clubs or other program activities and end up providing child 
care provider “tips,” can, in turn, change the child care environments for welfare recipients’ chil- 
dren. This section briefly summarizes program effects on the nontargeted outcomes for which 
data are available in the NEWWS Evaluation. Again, the effects on nontargeted outcomes are 
measured by comparing outcomes for the program and control groups, and differences in impacts 
for the client survey and COS samples are highlighted. 

• Some programs led to a reduction in health insurance coverage for both 
children and parents. 

In total, four programs — Riverside LFA, Portland, Columbus Integrated, and Oklahoma 
City — decreased health care coverage levels (as reported by parents) as of the end of the follow- 
up period.'® (See Table 3, which shows these effects for several different samples for which child 
impacts are discussed later in this document.) Portland program impacts were not statistically 
significant, but were just above the 10 percent level used as the standard throughout the NEWWS 
Evaluation analyses. For the client survey sample, the other seven programs had no impacts on 
health care coverage rates for children or parents. COS sample members in the Grand Rapids and 
Riverside LFA programs also reported a decrease in coverage, for focal children as well as any 
other children. (See Box 3 for a discussion of the dynamics that may have led to health insurance 
coverage reductions.) 



'*As noted earlier, the two-year follow-up period for many of the programs predated significant expansions in 
the Medicaid program and the establishment of the Children’s Health Insurance Program (CHIP). 
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National Evaluation of Welfare-to-Work Strategies 
Table 3 

Impacts on Selected Nontargeted Outcomes, by Sample 



Client Survey Sample: 

Families with All 

Client Survey Sample School-Age Children COS Sample 

Control Control Control 



Site 


Group 


Impact 


Group 


Impact 


Group 


Impact 




Adults and children have health care coverage at the end of year 2 (%) 


Atlanta LFA 


80.7 


-0.9 


76.8 


1.8 


90.5 


1.8 


Grand Rapids LFA 


80.4 


-3.1 


75.4 


-0.8 


88.7 


-2.8 * 


Riverside LFA 


84.7 


-3.9 ** 


79.8 


-1.6 


92.3 


-6.8 *** 


Portland 


85.6 


-5.1 


81.0 


-3.7 






Atlanta HCD 


80.7 


-1.0 


76.8 


0.0 


90.5 


1.2 


Grand Rapids HCD 


80.4 


-1.1 


75.4 


-1.5 


88.8 


-0.2 


Riverside HCD 


85.4 


-2.1 


81.3 


-7.1 ** 


91.0 


2.3 


Columbus Integrated 


80.9 


-7.1 ** 


79.2 


-8.8 ** 






Columbus Traditional 


80.9 


1.0 


79.2 


-0.7 






Detroit 


88.3 


-0.6 


81.6 


-4.7 






Oklahoma City 


67.6 


-10.9 ** 


61.6 


-14.3 * 










Used paid child care while employed (%) 






Atlanta LFA 


19.7 


5.8 *** 


11.2 


3.5 


29.4 


-5.7 


Grand Rapids LFA 


32.3 


7.4 *** 


17.9 


4.8 


41.7 


6.4 


Riverside LFA 


20.9 


7.9 *** 


13.6 


3.4 


17.4 


10.7 *** 


Portland 


29.4 


11.9 *** 


10.3 


12.8 ** 






Atlanta HCD 


19.7 


4.2 ** 


11.2 


3.8 * 


29.3 


1.0 


Grand Rapids HCD 


32.3 


-0.3 


17.9 


-4.0 


41.7 


-3.9 


Riverside HCD 


15.2 


6.6 *** 


10.5 


2.3 


11.9 


11.8 *** 


Columbus Integrated 


22.7 


5.5 * 


12.4 


4.3 






Columbus Traditional 


22.7 


2.4 


12.4 


3.4 






Detroit 


22.9 


13.0 *** 


7.2 


11.2 ** 






Oklahoma City 


29.5 


6.6 * 


15.5 


-1.2 







Used transitional child care benefits (%) 



Atlanta LFA 


5.3 


6.8 *** 


3.7 


0.7 Not calculated 


Grand Rapids LFA 


2.1 


2.9 *** 


2.0 


2.8 * 


Riverside LFA 


1.5 


1.9 *** 


2.9 


5.2 *** 


Portland 


12.5 


11.0 *** 


4.7 


8.7 ** 


Atlanta HCD 


5.3 


2.4 ** 


3.7 


-1.7 


Grand Rapids HCD 


2.1 


0.9 


2.0 


-0.4 


Riverside HCD 


1.0 


0.2 


1.3 


2.3 


Columbus Integrated 


3.9 


1.5 


1.8 


2.8 


Columbus Traditional 


3.9 


1.0 


1.8 


2.5 


Detroit 


2.1 


2.5 


0.8 


-0.4 


Oklahoma City 


14.0 


-2.4 


5.6 


-0.1 



(continued) 



O 




- 22 - 



31 



Table 3 (continued) 



Client Survey Sample: 
Families with All 



Client Survey Sample School-Age Children COS Sample 



Site 


Control 

Group 


Impact 


Control 

Group 


Control 
Impact Group 


Impact 






Had a babv since study entry 






Atlanta LFA 


6.4 


0.5 


3.0 


2.6 ** 


11.5 


-2.2 


Grand Rapids LFA 


11.1 


1.9 


4.6 


1.1 


17.6 


0.4 


Riverside LFA 


12.7 


-0.2 


7.1 


-0.2 


18.8 


-0.9 


Portland 


10.7 


-1.2 


3.7 


-0.1 






Atlanta HCD 


6.4 


1.4 


3.0 


0.8 


11.4 


2.6 


Grand Rapids HCD 


11.1 


2.4 


4.6 


-0.7 


17.6 


1.4 


Riverside HCD 


13.6 


0.7 


7.4 


0.1 


19.3 


-0.6 


Columbus Integrated 


7.9 


1.7 


3.2 


3.2 






Columbus Traditional 


7.9 


-3.2 * 


3.2 


-1.0 






Detroit 


12.3 


-2.6 


5.7 


-0.1 






Oklahoma City 


14.9 


0.7 


1.3 


2.6 










Married and living with spouse (%) 






Atlanta LFA 


4.0 


-0.3 


4.6 


-1.0 


4.3 


-0.2 


Grand Rapids LFA 


11.8 


1.3 


12.4 


2.5 


14.4 


-1.9 


Riverside LFA 


13.4 


-2.7 * 


15.9 


-3.4 


11.4 


-1.6 


Portland 


9.0 


-0.2 


7.6 


2.2 






Atlanta HCD 


4.0 


-1.2 


4.6 


-2.1 * 


4.3 


-0.8 


Grand Rapids HCD 


11.8 


0.3 


12.4 


2.0 


14.4 


-3.3 


Riverside HCD 


10.9 


1.6 


15.0 


-0.6 


7.2 


2.8 


Columbus Integrated 


9.0 


1.1 


10.0 


-1.6 






Columbus Traditional 


9.0 


0.9 


10.0 


-1.1 






Detroit 


7.6 


-3.4 


10.3 


-4.5 






Oklahoma City 


19.1 


-3.4 


25.7 


-9.9 * 










Single and 


living only with children 






Atlanta LFA 


58.1 


4.2 * 


55.6 


5.4 * 


Not calculated 


Grand Rapids LFA 


52.8 


-0.7 


54.4 


0.3 






Riverside LFA 


47.0 


2.4 


47.0 


2.1 






Portland 


52.0 


-6.7 


58.4 


-7.4 






Atlanta HCD 


58.1 


2.9 


55.6 


4.4 






Grand Rapids HCD 


52.8 


-0.2 


54.4 


-4.2 






Riverside HCD 


47.3 


-0.5 


44.8 


1.6 






Columbus Integrated 


55.1 


-0.1 


55.8 


0.5 






Columbus Traditional 


55.1 


-4.4 


55.8 


-4.7 






Detroit 


61.9 


1.5 


60.6 


1.4 






Oklahoma City 


41.2 


5.5 


38.2 


16.6 ** 







(continued) 
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Table 3 (continued) 



SOURCES: MDRC and Child Trends calculations from the Two-Year Client Survey. 

NOTES: For the COS sample, health insurance coverage and child care use are for focal children only. In 
addition, child care for focal children was not necessarily "paid.** 

Riverside limited enrollment in its HCD program to individuals determined by program regulations 
to need basic education, because they lacked a high school diploma or GED certificate, attained low 
scores on a reading or math test administered at program entry, or had limited proficiency in English. As 
a result, control group means differ for the Riverside LFA and HCD programs. 

A two-tailed t-test was applied to differences between outcomes for the program and control groups. 
Statistical significance levels are indicated as * = 10 percent, ** = 5 percent, and *♦* = 1 percent. 
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Box 3: How Welfare-to-Work Programs Can Lead 
to a Reduction in Health Insurance Coverage 

At study entry, almost all NEWWS Evaluation sample members and their de- 
pendent children had health coverage because they were receiving welfare and were 
automatically covered under Medicaid. (In Oklahoma City, where applicants for assis- 
tance whose eligibility was not yet determined were included in the sample, initial cover- 
age rates were lower.) Over time, coverage rates declined for both program and control 
group members, as some people left welfare and did not replace their Medicaid coverage 
with coverage from employers or other sources. By the end of the two-year follow-up pe- 
riod, between 81 percent (Columbus) and 88 percent (Detroit) of control group members 
reported having health care coverage for themselves and their children. (This range covers 
all sites except Oklahoma City, where the rate was 68 percent.) 

Three programs — Riverside LFA, Portland, and Columbus Integrated — that in- 
creased employment and decreased welfare receipt as of the end of the two-year follow-up 
also decreased health care coverage levels (as reported by parents). (No program in- 
creasedht&Wh insurance coverage.) Reductions in coverage ranged from 4 to 7 percentage 
points for sample members and their children, and from 3 to 6 percentage points for chil- 
dren only. Although many program group members who left welfare (and automatic 
Medicaid coverage) found a job that provided health insurance, received transitional 
Medicaid benefits, or obtained alternative sources of coverage, others were not able to re- 
place the coverage they had under Medicaid. Some of these individuals never received 
transitional Medicaid, and others had exhausted or had not restarted their benefits as of the 
end of the two-year follow-up period. 

Program group members in Oklahoma City reported even larger decreases in cov- 
erage: 1 1 percentage points for adults and children and 9 percentage points for children 
only. This program decreased welfare receipt and appears to have increased short-term 
employment that did not provide any health insurance. 



• The programs differed in their messages and practices concerning child 
care. 

All 1 1 programs offered child care assistance to welfare recipients who needed this ser- 
vice while they were participating in program activities or while they were employed. In the 



”Both access to and allowable payments for child care were the same for control and program group members 
in the NEWWS evaluation. If a control group member enrolled on her own in a community education or training 
activity or become employed, she would have been entitled to the same type of child care assistance that a program 
group member would have received. If a state would only pay for licensed child care, for example, then only this 
type of care would be paid for either control or program group members. Program group members, however, proba- 

(continued) 
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Atlanta LFA and HCD, Oklahoma City, Portland, and Detroit programs child care assistance was 
emphasized either by site staff or by the welfare department’s organizational structure. In both 
Atlanta programs case managers actively promoted the availability of child care reimbursement 
as a benefit of program participation. In Oklahoma City state-wide emphasis on access to child 
care made assistance to clients readily available while they were in the program and after they 
left welfare for work. Oklahoma City had no set caps on the amount of child care assistance that 
clients could receive. Atlanta and Oklahoma City, unlike the other programs, reimbursed only for 
care given by licensed providers. In Portland case managers did not push specific types or loca- 
tions of providers, but they did emphasize the necessity for individuals to make arrangements 
and assisted those who were unable to make arrangements on their own. In Detroit case manag- 
ers placed a priority on making child care payments, and made referrals to licensed providers in 
the area on request, but the choice of provider (including choosing licensed or imlicensed care) 
was left to the parent. 

Both the Grand Rapids and Columbus programs would reimburse expenses fi'om child 
care in licensed as well as unlicensed care, but expected parents to make their own arrangements. 
Referrals to licensed providers in the area could be made at the parents’ request. 

Child care providers were not difficult to come by in any site except Riverside, where 
case managers encouraged the use of low-cost, more informal arrangements, both to contain pro- 
gram costs and because case managers believed that parents would be more able to afford such 
arrangements after program or other government supports expired. 

The authorization of transitional child care benefits for those who left welfare for work 
was easiest in the Detroit, Portland, Columbus, and Oklahoma City programs. Few individuals 
who went to work in the other six programs received these benefits.^® 

• Most programs increased the use of paid child care during employment. 

Between 1 5 and 32 percent of control group members (depending on the site) in all 1 1 
programs used paid child care while employed at some point during the two-year follow-up pe- 
riod. Nine programs — all four of the employment-focused programs and five of the seven edu- 
cation-focused programs — produced impacts on paid child care, with the impacts in two pro- 
grams (Portland and Detroit) above 10 percentage points in magnitude. (See Table 3.) Relatively 
few control group members (less than 15 percent in any site) used transitional child care benefits. 
Five programs increased the use of such benefits, but these impacts were sizable only in the At- 
lanta LFA and Portland programs, where the increases in the receipt of these benefits were 7 and 
1 1 percentage points, respectively. Among COS sample members, only the two Riverside pro- 
grams increased child care during employment for the focal child (measured during the month 



bly heard messages about the importance of child care and the advantages of particular types of child care more 
frequently than did control group members, owing to program group members’ increased exposure to caseworkers 
and other program-related staff. 

^°For more information on the methods of making child care payments, reimbursement rates, and child care 
allowances to assist welfare recipients who had earned income in the Atlanta, Grand Rapids, Riverside, and Portland 
sites, see Hamilton et al., 1997, and Scrivener et al., 1998. 
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prior to the client survey interview), and this care tended to be informal and to occur during non- 
standard hours (evenings or weekends). Thus, in no site were COS focal children more likely 
than their control group counterparts to be in formal child care during their mother’s employ- 
ment. (See Box 4 for a discussion of how child care use and employment might have interacted.) 



Box 4; Most Programs Increased the Use of Child Care 
During Employment Independent of Their Effects 
on Increasing Employment 

As noted, nine programs produced increases in the use of paid child care while em- 
ployed, ranging from 4 to 13 percentage points. Increases in employment do not entirely ex- 
plain program impacts on child care use and on paid care while employed, since in many pro- 
grams impacts on child care use were maintained even when only those who worked during the 
follow-up period are considered (a nonexperimental comparison). A likely explanation is that 
employed program group members required or preferred more stable child care arrangements 
than employed control group members, either because of the different characteristics of their 
jobs or because they more frequently heard messages from case workers regarding the impor- 
tance of child care. (Keep in mind, as noted earlier, that practices related to child care assis- 
tance — access to and allowable payments for child care — were the same for control and pro- 
gram group members within each site.) For example, case managers in the Atlanta LFA pro- 
gram (as well as in the Atlanta HCD program) encouraged people to use child care and empha- 
sized it as a reason to participate in the program, in part by strongly emphasizing the availabil- 
ity of reimbursement for child care costs. For the client survey sample, the LFA program did 
not increase employment levels but did increase child care use while employed by 4 percentage 
points and raised use of paid child care by nearly 6 percentage points. Similarly, case managers 
in Detroit placed a priority on arranging child care, and the increase in the use of paid child 
care (13 percentage points) exceeded the program’s employment gains. 



* The programs had little, if any, effect on fertility or family structure or 
living arrangements. Any effects found were small and not clustered 
among certain programs. This was true for the client survey sample as 
well as for the COS sample. 

Across all 1 1 programs, only Columbus Traditional had any effect on fertility; this pro- 
gram resulted in a 3 percentage point decrease in the proportion of sample members who had a 
baby since study entry. (See Table 3.) Only two programs had impacts on marital status: in Port- 
land a 5 percentage point increase in the proportion of sample members living as an unmarried 
couple (not shown in Table 3), and in Riverside LFA a 3 percentage point reduction in the pro- 
portion of sample members married and living with a spouse. Similarly, only two programs had 
impacts, small in size, on family household composition. Impacts on housing status were not ex- 
tensive either, although five programs did produce impacts (not shown in Table 3). The largest of 
these was a 5 percentage point increase in the proportion of program group members in the Co- 
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lumbus Traditional program who lived with family or friends and paid rent. For COS sample 
members, none of the programs had an impact on fertility or marital status, and only Riverside’s 
LFA program had an impact on families’ living arrangements, with fewer fathers living with 
their biological focal child. 

• The few effects on mothers’ psychological functioning and stress or on 
parenting can be considered small. These particular nontargeted out- 
comes were examined only for the COS sample. 

Measures specific to the COS component of the two-year client survey were used to as- 
sess mothers’ psychological well-being and parenting. Psychological well-being was assessed in 
three areas: time stress, locus of control or self-efficacy, and depression. Increases in mothers’ 
feelings of time stress were found in both Atlanta programs and in the Riverside HCD program. 
There were no impacts on mothers’ feelings of control over their own lives. Only the Grand Rap- 
ids LFA program had an impact — an unfavorable one — on mothers’ depressive symptoms. 
Parenting was assessed on such dimensions as maternal warmth, maternal aggravation, and ma- 
ternal cognitive stimulation. Impacts were found on two of the parenting measures for the At- 
lanta HCD program (both favorable), on four of the measures for the Atlanta LFA program (all 
favorable), and on one measure for the Grand Rapids LFA program (unfavorable). No impacts on 
parenting were found for the other programs. (The impacts discussed in this paragraph are not 
shown in a table.) 

VIII. Child Impacts 

This section summarizes impacts for children, using data fi"om the Child Outcomes Study 
(COS) sample in three sites and from the client survey sample in all seven evaluation sites. COS 
data provide a rich, in-depth look at a subset of young children; child data from the client survey 
sample, though more limited, cover more sites and programs, provide information about a large 
number of children who were age 6 or over at study entry, and are available for families in four 
sites who had children as young as age 1 at baseline. (See Appendix C for a discussion of how 
the children in the control groups in this study and national samples of children compared devel- 
opmentally at the two-year follow-up point.) Given, however, that many of the client survey 
questions applied only to school-age children, most analyses reported below narrow the client 
survey sample to only those who had all school-age children at study entry.^' (Results for client 
survey sample members with children of all ages are shown in Appendix D.) Child impacts, as 
was the case with the previously discussed impacts, are measured by comparing outcomes for 
children of program group members with outcomes for children of control group members. Child 



^‘For example, parents with no school-age children would have to respond “no” when asked if any of their chil- 
dren had been suspended from school. As mentioned earlier, approximately 49 percent of the parents in the client 
survey sample had all school-age children. 
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impacts are presented for three child outcome areas: behavioral and emotional adjustment, cogni- 
tive functioning and academic achievement, and health and safety.^^ 

A. Spillover Effects of Welfare-to-Work Programs on Children’s Behavioral 
and Emotional Adjustment 

• For the young children in the COS sample, program impacts on behav- 
ioral and emotional adjustment were infrequent and both favorable and 
unfavorable. 

For focal children in the COS sample, indicators of behavioral and emotional problems 
included the Behavior Problems Index (BPI) and the Positive Behavior Scale/Social Competence 
Subscale (PBS/SCS). For each of the six COS programs, 1 1 outcome measures were developed 
from these two instruments. Two of the six programs had any impacts on focal children in this 
outcome area, producing a total of five impacts. (See Table 4.) Two of the five impacts were in 
the Grand Rapids LFA program; they were unfavorable and they related to the BPI. The three 
remaining impacts were in the Atlanta LFA program, and they were both favorable and unfavor- 
able and related to the BPI and the PBS/SCS. 

• For school-age children across all the evaluation sites, 8 of the 11 pro- 
grams produced at least one impact on behavioral and emotional adjust- 
ment These effects were both favorable and unfavorable. 

In the client survey sample, children’s behavioral and emotional adjustment was meas- 
ured by asking parents whether their children (1) had been suspended from school, (2) were re- 
ceiving or requiring help for behavioral or emotional problems, or (3) were in a special class or 
school for such problems. Among the subgroup of families with all school-age children, for 
whom these measures would be directly applicable, between 22 and 35 percent (depending on the 
site) of the control group parents reported that at least one of their children had been suspended 
from school since study entry; between 16 and 45 percent (depending on the site) reported that at 
least one child was currently receiving or requiring help for behavioral or emotional problems; 
and between 6 and 17 percent (depending on the site) reported that they had a child (or children) 
attending a special class or school for behavioral or emotional problems. (See Table 5.) Eight of 
the 11 programs produced at least one impact on children in this outcome area. (See Table 6.) 
Three programs decreased the incidence of at least one behavioral problem, and five programs 
increased the frequency of at least one. Only two programs, however, had an impact on more 
than one of the three behavioral adjustment measures. 



presenting results for young children, this section discusses impacts on the “focal” child in the COS fami- 
lies in the three sites in which this study was embedded, not impacts on the focal child’s siblings, most of whom 
were older. Here, impacts on these siblings are included with those of all school-age children or with those of chil- 
dren of all ages across the seven evaluation sites (depending on the child outcome measure). McGroder et al., 2000, 
presents child impacts for the focal child as well as the focal child’s siblings. Summary statements of child impacts 
in that report often combine all children in the COS families and thus may differ somewhat from the “yoxmg child” 
summary statements in this document. 
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National Evaluation of Welfare-to-Work Strategies 
Table 4 

For Focal Children in the Child Outcomes Study Sample: Impacts on Child Outcomes 

Behavioral.and Emotional Adjustment: 



Behavior Problems Index (BPI) Positive Behavior Scale 

Externalizing Behavior Internalizing 
Total Behavior Problems Problems Behavior 





At or 


At or 




At or 


At or 


At or 


At or 


At or 




Below 


Above 




Below 


Above 


Below 


Below 


Above 




25th 


75th 




25th 


75th 


75th 


25th 


75th 




Percentile 


Percentile 




Percentile 


Percentile 


Percentile 


Percentile 


Percentile 


Site and Program Mean 


(%) 


(%) 


Mean 


(%) 


(%) 


Mean (%) 


Mean (%) 


(%) 


Atlanta LEA 


U 




F 










F 


Grand Rapids LFA 
Riverside LFA 






U 


U 










Atlanta HCD 
Grand Rapids HCD 
Riverside HCD 



















Cognitive Functioning and Academic Achievement: 





Bracken School Readiness Component 


Had Academic Problems (%) 


Site and Program 


At or Below 25th 

Age- Adjusted Mean Score Percentile (%) 


At or Above 75th 
Percentile (%) 


Atlanta LFA 


F F 


F 


Grand Rapids LFA 






Riverside LFA 






Atlanta HCD 




F 


Grand Rapids HCD 




F 


Riverside HCD 







Health and Safety: 



General Health Rating 


Had Emergency Hospital Visit (%) 


In "Very Good" or 





Site and Program 


Mean 


Excellent Health (%) 


Atlanta LFA 
Grand Rapids LFA 
Riverside LFA 


U 


U 


Atlanta HCD 
Grand Rapids HCD 
Riverside HCD 


U 


U 



SOURCE: Child Trends calculations from the Two-Year Client Survey. 

NOTES: Impacts shown are only those statistically significant at the 10 percent level or above, using a two-tailed t-test. ”F” 
indicates a favorable impact estimate; "U" indicates an imfavorable impact estimate. 




-30- 



39 



National Evaluation of Welfare-to-Work Strategies 
Table 5 



For Families with All School-Age Children: 
Control Group Child Outcomes 



Site and Program 




Behavioral Adjustment 


School Progress 


Health and Safety 


Suspended 
Sample from School 
Size (%) 


Receiving or 
Requiring Help 
for Behavioral 
or Emotional 
Problems 
(%) 


Attends a 
Special 
Class for 
Behavioral 
Problems 
(%) 


Repeated a 
Grade 
(%) 


Attends a 
Special 
Class for 
Learning 
Problems 
(%) 


Removed 

from 

Mother's 

Care 

(%) 


Taken to 
Hospital for 
Accident, 
Injury, or 
Poisoning 
(%) 


Atlanta 


549 


29.6 


19.7 


9.3 


19.2 


14.2 


3.2 


20.9 


Grand Rapids 


253 


25.7 


34.4 


13.8 


14.4 


31.6 


4.5 


32.0 


Riverside LFA 


592 


21.6 


25.1 


6.1 


11.4 


23.6 


3.6 


29.0 


Riverside HCD 


385 


26.4 


21.4 


6.4 


14.4 


20.8 


4.1 


24.3 


Columbus 


187 


35.1 


27.4 


14.1 


22.0 


31.2 


1.9 


28.3 


Detroit 


84 


34.7 


16.1 


6.2 


19.0 


17.1 


1.4 


12.5 


Oklahoma City 


83 


26.4 


17.5 


8.2 


22.6 


32.5 


4.5 


33.1 


Portland 


118 


33.7 


44.5 


16.6 


7.7 


29.2 


9.3 


29.9 



SOURCE: MDRC calculations from the Two-Year Client Survey. 
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National Evaluation of Welfare-to-Work Strategies 
Table 6 
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NOTES: A two-tailed t-test was applied to differences between outcomes for the program and control groups. 
Statistical significance levels are indicated as * = 10 percent, ** = 5 percent, and *** = 1 percent. 



B. Effects of Welfare-to-Work Programs on Children’s Cognitive Functioning 
and Academic Achievement 

• For the young children in the COS sample, favorable program impacts on 
cognitive development were found, although they can be considered small 
in magnitude. Impacts were concentrated in the Atlanta LFA program. 

For focal children in the COS sample, the Bracken Basic Concept Scale/School Readi- 
ness Composite (BBCS/SRC) was used to measure cognitive development. For each of the six 
COS programs, three outcome measures were developed from the BBCS/SRC scores (reflecting 
the average score and the distribution of scores), and one outcome measure consisted of an index 
of two survey questions asked of mothers about academic problems. Three of the six programs 
had impacts in this outcome area, and they were all in a desirable direction. (See Table 4.) The 
Atlanta LFA program increased the average BBCS/SRC score, increased the proportion of focal 
children scoring in the top quartile, and decreased the proportion scoring in the bottom quartile. 
The Atlanta and Grand Rapids HCD programs slightly increased the proportion of children scor- 
ing above the 75th percentile, but the mean score was imaffected. 

• For school-age children across all evaluation sites, few program impacts 
on academic achievement were found. 

Children’s school progress was measured by asking parents whether their children (1) had 
repeated a grade or (2) were attending a class for learning problems. Between 8 and 23 percent 
(depending on the site) of control group families with all school-age children reported that at 
least one of their children had repeated a grade in school during the two-year follow-up period; 
between 14 and 33 percent reported that any of their children were currently attending a special 
class for learning problems. (See Table 5.) For program group families with all school-age chil- 
dren, two programs — Riverside LFA and Columbus Integrated — had any impacts in this area; 
again, these were in a desirable direction. (See Table 6.) 

C. Effects of Welfare-to-Work Programs on Children’s Health and Safety 

• While most mothers in the COS sample rated their children as being in 
very good or excellent health, in one site the welfare-to-work programs 
resulted in a decrease in this rating. 

In the area of health and safety, mothers were asked to rate their focal child’s overall 
health by answering the question: “Would you say that your child’s health in general is excellent, 
very good, good, fair, or poor?” For each of the six COS programs, two outcome measures were 
developed from the responses. In addition, a third outcome measure concerned whether the focal 
child had had a serious accident, injury, or poisoning. Between 77 percent (Atlanta) and 82 per- 
cent (Riverside) of control group mothers in the COS sample reported that their focal child was 
in very good or excellent health; between 13 percent (Atlanta) and 24 percent (Grand Rapids) 
reported that their focal child had had a serious accident, injury, or poisoning since study entry. 

Two of the six programs — Riverside LFA and HCD — had impacts in this area. In both 
of these programs, the foimd impacts, which were based on the mother’s health rating, were im- 
favorable: Mothers’ average rating of the general health of their focal child decreased slightly. 
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and a smaller proportion of children were reported by mothers to be in very good or excellent 
health. (See Table 4.) Specifically, as a result of the Riverside LFA and HCD programs, mean 
health ratings (on a scale of 1 to 5 points) decreased by .23 and .20 points, respectively; the pro- 
portion of focal children rated as in very good or excellent health decreased by about 12 and 10 
percentage points and the proportion rated as in fair or poor health increased by 5 and 4 percent- 
age points, respectively. 

It should be noted that the primary measure here was one of global health and not specific 
health problems. In addition, the health assessments were made by mothers, and not by impartial 
doctors or through a review of health records. While it is entirely possible that the Riverside pro- 
grams truly changed children’s health status, it is also possible that these findings reflect changes 
in mothers’ perceptions of their children’s health. As discussed earlier, the Riverside LFA and 
HCD programs produced large increases in the likelihood that COS mothers would be employed 
at some point during the two-year follow-up period; their two-year employment rates, relative to 
control groups’, increased by 65 percent in the Riverside LFA program and by 45 percent in the 
Riverside HCD program. The next largest increase in two-year employment rates was a 15 per- 
cent increase, achieved in the Grand Rapids LFA COS sample. It is possible that mothers in the 
Riverside LFA and HCD COS samples, given their much greater likelihood of employment, per- 
ceived their focal children as being in poorer health than mothers in the control groups. For these 
mothers, even relatively minor focal child health problems (e.g., ear infections) could have 
caused disruptions in their daily lives, because they would have needed to either stay home from 
work to care for the child or perhaps quickly make alternative, non-group child care arrange- 
ments so they could go to their jobs. Among control group mothers, who were much less likely 
to be employed, these same relatively minor health problems might not have been as disruptive 
and thus memorable. 

• For all children across the evaluation sites, impacts on the likelihood of 
events suggesting other child health or safety issues were not common. 

Children’s health and safety was measured by asking parents if any of their children (1) 
had been removed from their care or (2) had had a serious accident, injury, or poisoning. Among 
all control group parents in the seven sites, a surprisingly high proportion — up to 8 percent in a 
site — reported that a child had been removed from their care during the two-year follow-up pe- 
riod because they could not care for or handle the child. (See the upper panel of Table D.l.) Be- 
tween 18 and 37 percent of all control group members reported that during the previous two 
years at least one of their children had had an accident, injury, or poisoning requiring a visit to a 
hospital emergency room or clinic. Among control group parents with all school-age children, 
these statistics were similar. (See Table 5.) When all families are considered, one of the 1 1 stud- 
ied programs (Coliunbus Traditional) had an impact on children’s being removed from their 
mother’s care (an increase in the incidence of this event) and no programs affected the likelihood 
of children having an accident, injury, or poisoning that required immediate medical attention. 
(See the lower panel of Table D.l.) When only families with all school-age children are consid- 



^This does not necessarily mean that the child was placed in the foster care system; an unruly teenager, for ex- 
ample, could have been sent to live temporarily with a relative. 
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ered, two programs — Columbus Traditional and Grand Rapids HCD — increased the incidence 
of children being removed from their home and no programs had an impact on the latter out- 
come. (See Table 6.) 

D. Clustering of Impacts by Child Outcome Area 

• For the young children in the COS sample, impacts (favorable or unfa- 
vorable) were not clustered (that is, concentrated) in a particular child 
outcome area. 

Two programs had at least one impact on behavioral and emotional adjustment measures; 
three programs had at least one impact on cognitive functioning and academic achievement 
measures; and two programs had any impacts on health and safety measures. The 14 impacts 
found were about evenly split among the three child outcome areas. (See Table 7 for a summary 
of the COS focal child impacts.) Notably, however, the behavioral and emotional adjustment im- 
pacts were both favorable and unfavorable; all of the cognitive functioning and academic 
achievement impacts were favorable; and the health and safety impacts were unfavorable. There 
is some evidence that suggests that the diverging directions of the impacts on behavioral out- 
comes for these young children may reflect the fact that some of the programs affected underly- 
ing processes, such as parenting, in different ways. 

• For school-age children across all evaluation sites, impacts tended to be 
clustered in the behavioral adjustment area; relatively few impacts were 
found in the areas of school progress or health and safety. 

For this group of families, eight programs had at least one impact on children’s behav- 
ioral and emotional adjustment; only two programs had any impacts on either academic progress 
or health and safety. As was the case for the young children in the COS sample, the behavioral 
and emotional impacts were both favorable and unfavorable; the few impacts on academic pro- 
gress were favorable; and the few impacts on health and safety (both concerning removal of a 
child from a mother’s care) were unfavorable. 

E. Clustering of Impacts by Program 

• For the young children in the COS sample, impacts were somewhat clus- 
tered (that is, concentrated) in the Atlanta LFA program. 

For focal children of mothers subject to this program, there was a decrease in the propor- 
tion of children scoring at the low end on the Behavior Problems Index (BPI) — the one unfa- 
vorable impact of the Atlanta LFA program. All other impacts for this program were favorable: 
There was a decrease in the average frequency of BPI externalizing behavior or emotional prob- 
lems and an increase in the proportion of children scoring at the high end on the Positive Behav- 
ior Scale. In addition, there was an increase in the average Bracken School Readiness Composite 
test score, a decrease in the proportion of children scoring at the low end of the test, and an in- 
crease in the proportion scoring at the high end. (See Tables 4 and 7.) In contrast, other programs 
had two child impacts at most: In both Riverside programs, mothers’ average rating of the gen- 
eral health of their focal child decreased slightly, and a smaller proportion of children were re- 
ported by their mother to be in very good or excellent health. 
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National Evaluation of Welfare-to-Work Strategies 
Table 7 

For Focal Children in the ChUd Outcomes Study Sample: 
Number of Impacts on ChUd Outcomes 



Site and 
Program 


Behavioral and 
Emotional 
Adjustment 


Cognitive Functioning 
and Academic 
Achievement 


Health and 
Safety 


Total 
All Areas 


Atlanta LFA 


3 (2F/1U) 


3 F 




6 (5F/1U) 


Grand Rapids LFA 


2 U 






2 U 


Riverside LFA 






2 U 


2 U 


Total number of impacts 


5 (2F/3U) 


3 F 


2 U 


10 (5F/5U) 


Number of possible impacts 


33 


12 


9 


54 


Atlanta HCD 




1 F 




1 F 


Grand Rapids HCD 




1 F 




1 F 


Riverside HCD 






2 U 


2 U 


Total number of impacts 


0 


2 F 


2 U 


4 (1F/3U) 


Number of possible impacts 


33 


12 


9 


54 


All 6 programs 


5 (2F/3U) 


5 F 


4 U 


14 (7F/7U) 


Number of possible impacts 


66 


24 


18 


108 



SOURCE: Child Trends calculations from the Two-Year Client Survey. 



NOTES: Impacts shown are only those statistically significant at the 10 percent level or above, using a 
two-tailed t-test. indicates a favorable impact estimate; ”U” indicates an unfavorable impact 
estimate. 
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• For school-age children across all evaluation sites, impacts were not clustered in 
particular programs. 

As noted earlier, seven child outcomes were measured for children in this age group. At 
most, impacts were found on three of the outcomes in any given site; The Riverside LFA pro- 
gram increased the proportion of parents reporting that a child had been suspended from school 
during the previous two years or that a child was attending a special class for behavioral or emo- 
tional problems and decreased the proportion of parents reporting that a child had repeated a 
grade in the past two years. (See Table 6.) Thus, impacts for this program were both unfavorable 
and favorable. 

F. Size of Impacts 

Few evaluations of welfare-to-work programs that have examined effects on children 
used a random assignment research design. As a result, in contrast to the situation for adult im- 
pacts, few benchmarks for characterizing the magnitude of child impacts exist. Nevertheless, this 
section attempts to assess the size of the found child impacts. 

• For the young children in the COS sample taken as a group, almost all 
child impacts can be considered small in magnitude. 

One of the child impacts for this sample — the decrease in the proportion of mothers in 
the Riverside LFA program rating their young child as in excellent or very good health — had an 
effect size of one-third of a standard deviation. All other young child impacts were of a smaller 
magnitude, although, as will be discussed below, a few of the child impacts for lower-risk sub- 
groups were larger. For example, the increase in the average Bracken School Readiness Compos- 
ite test score for children of mothers in the Atlanta LFA program represents an increase of .14 of 
a standard deviation and indicates that focal children of program group mothers knew, on aver- 
age, alm ost two more school readiness concepts than focal children of control group mothers. (A 
total of 61 concepts are assessed in this test.) 

• For school-age children across all evaluation sites, most of the child im- 
pacts can be viewed as small, but some are clearly larger and some are of 
concern. In two sites, for example, there was an increase in the proportion 
of parents reporting that a child had been removed from their care. 

Most of the child impacts for this sample can be judged as small. Some of the impacts, 
however, are not so small. For example, the Oklahoma City program increased by 17 percentage 
points the proportion of parents reporting that at least one of their children was currently receiv- 
ing or requiring help for behavioral or emotional problems (35 percent of program group parents 
reported this situation compared with 1 8 percent of control group parents).^'' Some of the im- 



is not clear what might have led to this child impact, the only one found in Oklahoma City. For families 
with all school-age children in the Oklahoma City sample, there were no impacts on two-year employment, earn- 
ings, or income. There was, however, a decrease in reported health care coverage, a decrease in the proportion of 
parents married and living with their spouse, and an increase in the proportion of parents who were single and living 

(continued) 
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pacts, though smaller in absolute size, are of concern owing to their nature. As one example, four 
programs increased the proportion of parents reporting that at least one of their children was cur- 
rently attending a special class for behavioral or emotional problems (although an additional two 
programs had impacts in the opposite direction on this measure). As another example, the Co- 
lumbus Traditional and the Grand Rapids HCD programs increased the proportion of parents re- 
porting that a child had been removed from their care during the two-year follow-up period be- 
cause they could not care for or handle the child. In the Colxunbus Traditional program, this in- 
crease was 6 percentage points (8 percent of program group parents compared with 2 percent of 
control group parents — a four- fold increase) and the Grand Rapids HCD program produced a 4 
percentage point increase here (9 percent of program group parents compared with 5 percent of 
control group parents).^^ 

It is unclear why there was an increase in the proportion of parents in two sites reporting 
that a child had been removed from their care. A similar result was foimd in a random assign- 
ment evaluation of the New Chance program, a voluntary demonstration project for young 
women who had children as teenagers and were high school dropouts.^^ The hypotheses that have 
been suggested for the New Chance finding, however — increased exposure to program staff, an 
increase in mothers moving out of their parental homes, and increased maternal depression — 
largely do not “hold true” for the Colxunbus Traditional and Grand Rapids HCD programs. While 
program group members in these two programs would have had more contact with case manag- 
ers than their control group coxmterparts, increasing the chances that child abuse problems might 
have been identified, they would not have had any increase in exposure to program staff relative 
to their coxmterparts in the Colxunbus Integrated and Grand Rapids LFA programs. Impacts on 
child removal from the home were not foxmd in these latter two programs.^^ In addition, few im- 



only with their children. (See Tables 2 and 3.) There are no clear connections between these findings and the in- 
crease in children’s receiving help for behavioral or emotional problems. 

^^The results of two quite conservative tests — the Tippet test and the Fisher test, developed in the literature on 
research synthesis (Cooper and Hedges, 1994) — indicate that at least one of these two impacts is, in fact, statisti- 
cally significant in view of the large number of programs examined. 

^^At a 42-month follow-up point. New Chance had increased the proportion of mothers living without any of 
their children as well as without at least one of their children. In addition, a higher proportion of children were in 
foster care. (See Quint et al., 1997, pp. 138-144.) New Chance researchers could not identify a clear reason for this 
finding, but hypothesize that increased exposure to program staff, resulting in a higher likelihood of identifying 
child welfare issues among program group mothers than control group mothers, might have played a role. In addi- 
tion, program group mothers were more likely to have moved out of their parents’ home, a situation that might have 
placed them more at risk for child neglect if there were no older family members around to help watch the children. 
Finally, the program did increase maternal depression, and this may have increased childrearing problems. 

^^It is possible that the provision of “on-site” child care centers in these programs, where mothers could drop off 
their children, played a role in this fmding. In both Grand Rapids and Columbus, some of the providers of job club 
and education activities (e.g., the public school system in Grand Rapids) operated such child care centers in the 
same building as the activities took place, staffed by well-trained child care workers, that mothers could use while 
they were participating in job club or attending education classes. Program administrators in Grand Rapids remem- 
ber at least a few cases where child care workers from these centers, knowing that most of the mothers of the chil- 
dren under their care were welfare recipients, alerted the welfare department to possible cases of child neglect or 
abuse. Although these centers would have been open to individuals in the Grand Rapids LFA as well as HCD pro- 
grams, and to those in the Columbus Integrated as well as Traditional programs, length of stay in the programs (and 
on welfare) was longer in the Grand Rapids HCD program than in the LFA program, and longer in the Columbus 

(continued) 
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pacts on family living arrangements, which might lead to child neglect or abuse, were found in 
the two programs. These impacts were the following: For families with children of all ages in the 
Columbus Traditional program, there was a 5 percentage point increase, noted earlier, in the pro- 
portion living with family or friends and paying rent. For the same group of families in the Grand 
Rapids HCD program, there was a 3 percentage point decrease in the proportion of families 
whose household included relatives and a 3 percentage point increase in the proportion of fami- 
lies whose household did not include the parent’s children. Finally, symptoms of maternal de- 
pression were examined in the Grand Rapids programs (not in the Columbus programs), and only 
for mothers in the COS sample. For that group of mothers in the Grand Rapids HCD program, 
however, no effect on mothers’ depressive symptoms was found (although an increase in such 
symptoms was found for COS mothers in the Grand Rapids LFA program). 

G. Balance Between Favorable and Unfavorable Impacts 

• Overall, for both the young children in the COS sample and the school- 
age children across all evaluation sites, program impacts on child devel- 
opment were as likely to be favorable as unfavorable. 

There were equal numbers of favorable and unfavorable impacts on the focal child. (See 
Table 7.) As previously noted, however, favorable impacts were concentrated in the area of cog- 
nitive development and the few health impacts found were unfavorable. Similarly, for school-age 
children in all 1 1 programs, there was an almost equal mix of favorable and unfavorable impacts. 
(See Table 6.) 

H. Variations in Impacts by Subgroup 

• Impacts were examined separately for the young children in the COS 
sample who, as of study entry, were at high risk or at low risk for poor 
development Few impacts were found within these subgroups. Among 
these, impacts on children at higher risk were small, and in two of the 
three sites tended to be favorable for education-focused programs and 
unfavorable for employment-focused programs; impacts on children at 
lower risk were larger, tended to be unfavorable, did not tend to vary by 
program approach, and were clustered in three programs. This type of 
subgroup analysis was not conducted for school-age children across all 
sites. 



Traditional program than in the Integrated program, thus increasing the length of time that children could have been 
in these child care centers and thereby the chances that a case of neglect or abuse would be identified and brought to 
the attention of the welfare department. The impacts on child removal from the home, however, are larger in the 
Grand Rapids HCD and Columbus Traditional samples that consist of families with all school-age children (who 
would have had less use of these child care centers), compared with the samples that consist of families with chil- 
dren of all ages, suggesting that while this particular hypothesis may explain part of these impacts, it is not a com- 
plete explanation. 
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In subgroup analyses examining child impacts for COS sample members, four subgroup 
divisions were defined, based on family baseline characteristics, that past research has suggested 
contain a higher-than-average proportion of children at risk for poor development. These over- 
lapping subgroups included families with three children or more or at least two children bom less 
than two years apart (the sibling configuration risk subgroup); families in which the mother, at 
baseline, did not have a high school diploma or GED or had low scores on reading or math tests 
(the educational risk subgroup); families in which the mother, at baseline, had received at least 
five years of welfare, reported at least four barriers to employment, or had never worked full time 
for six months or more for the same employer (the work risk subgroup); and families in which 
the mother, at baseline, reported symptoms of depression and a lack of control over her own life 
(the maternal psychological well-being risk subgroup). Within the COS control group, families 
who met the criteria of any of these, four subgroups had children who, as of the two-year follow- 
up point, were generally not developing as well as children in families who did not meet these 
criteria. 

Prior research, however, has suggested that the accumulation of risk may be more impor- 
tant than any particular risk factor for children’s development. While a child may be able to 
overcome a single risk factor, the accumulation of risk may “tip the scales” against a child, and 
result in unfavorable child outcomes. For the COS subgroup analysis, families who met the crite- 
ria of no subgroup or only one subgroup were considered to be in the lower cumulative risk sub- 
group; families who met the criteria of two, three, or all four of the subgroups were considered to 
be in the higher cumulative risk subgroup. Thus, all families were in one of these two subgroups. 

Relatively few young child impacts were found for each of the four defined overlapping 
subgroups and for the ciunulative risk subgroups. The impacts on focal children at higher risk for 
poor development were small, but in two of the three sites tended to be favorable for education- 
focused programs and unfavorable for employment-focused programs. The impacts on focal 
children at lower risk for poor development were larger, tended to be unfavorable, and did not 
tend to vary by program approach.^® The unfavorable impacts for focal children at lower risk 
were clustered in the Grand Rapids LFA program and in both of the Riverside programs. As 
noted earlier, this type of subgroup analysis was not conducted for school-age children across all 
evaluation sites. 

I. Impact Differences Between Employment- and Education-Focused Programs 

• For the young children in the COS sample, as vrell as for families vrith all 
school-age children in the 11 programs, child impacts were not systemati- 
cally different for mothers subject to employment-focused programs than 
for those subject to education-focused programs. 



“Note that statistical tests were not applied to the within-site differences in child impacts for the employment- 
focused programs as compared with the education-focused programs in this analysis. In addition, this analysis com- 
pared, for most of the subgroups in the Riverside site, individuals in the education-focused program — all of whom 
lacked a high school diploma or GED, had low literacy levels, or had limited English skills — with individuals in 
the employment-focused program, who may or may not have had these credentials or skills. 
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The strong experimental research design implemented in the Atlanta, Grand Rapids, and 
Riverside sites, comparing the effectiveness of LFA and HCD programs, permits a clear assess- 
ment of whether program approach — employment- or education-focused — explains any child 
impacts foimd. As noted earlier, among the yoimg children in the COS sample, 6 of the 14 child 
impacts foimd were within the Atlanta LFA (employment-focused) program. The Atlanta HCD 
(education-focused) program produced only one child impact. (See Table 4.) However, a pattern 
of more focal child impacts in LFA programs than in HCD programs did not occur in Grand 
Rapids and Riverside, the two other sites in which the COS was nested. In fact, the child impacts 
found for the two Riverside programs related to an identical child health measure and were unfa- 
vorable for both the LFA and HCD programs in Riverside. Thus, for the COS sample as a whole, 
child impacts were not consistently favorable or unfavorable in LFA or HCD programs either. As 
discussed earlier, however, the situation was somewhat different when subgroups within the COS 
sample were examined. 

Among families with all school-age children in the 11 NEWWS Evaluation programs, 
employment- and education-focused programs also did not produce systematically different child 
impacts. (See Table 6.) In the three evaluation sites (Atlanta, Grand Rapids, and Riverside) in 
which the research design permitted direct comparisons of these two program approaches, there 
were no clear differences in child impacts. Overall, for the client survey sample in the 11 pro- 
grams, child impacts were not clustered in one of the two types of program, and neither type of 
program had consistently favorable effects while the other had consistently unfavorable effects. 

J. Possible Explanations for the Few Child Impacts That Were Found 

Further research is needed to clearly determine the mechanisms through which some of 
the programs affected children. As noted above, the strong experimental research design imple- 
mented in three of the evaluation sites allows for a rigorous examination of whether program ap- 
proach affects child impacts. To examine whether other program features explain child impacts, 
however, nonexperimental approaches, which do not have the rigor of the experimental results 
discussed so far in the document, are required. At this point in the NEWWS Evaluation, two ap- 
proaches have been used. Possible patterns in child impacts were ascertained by taking advantage 
of the large number of programs studied in the school-age child analysis and simply assessing 
whether child impacts clustered according to the size of the 11 programs’ impacts on targeted 
and nontargeted outcomes. In addition, statistical mediational analyses were carried out for se- 
lected focal child impacts in the COS sample. Results from both approaches, which provide only 
suggestive explanations and do not indicate causality, are briefly summarized below. 

• Did programs with different sanctioning practices produce systematically 
different child impacts? For families with all school-age children in the 11 
programs, measured child impacts were not necessarily different for 
mothers subject to programs with moderate to high enforcement of the 
participation mandate than for mothers in programs with low enforce- 
ment 

There is no obvious relationship between the frequency with which programs imposed 
sanctions and the observed patterns of child impacts for the client survey sample. (See Table 8, 
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For Families with All School-Age Children: 

Impacts on Child Outcomes, Clustered by the Magnitude of Program 
Impacts on Sanctioning, Educational Attainment, Employment, and Income 

Behavioral Adjustment School Progress Health and Safety 

Receiving or 
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SOURCE: MDRC calculations from the Two-Year Client Survey. 
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which arrays the same child impacts shown in Table 6 according to the magnitude of each pro- 
gram’s impacts on various program implementation features and targeted outcomes. The child 
impacts are simply rearranged in each panel of the table.) For families with all school-age chil- 
dren, the two Columbus programs and the Atlanta HCD program had the highest sanction rates. 
Few child impacts were found for these three programs, with two favorable impacts for the Co- 
lumbus Integrated program; an unfavorable child impact (an increase in the incidence of a child 
being removed from the mother’s care) for the Columbus Traditional program; and no statisti- 
cally significant child impacts for the Atlanta HCD program. Among the four programs in which 
sanctioning rates were the lowest for families with all school-aged children, child impacts were 
both favorable and unfavorable. 

• Did programs that increased the likelihood of parents^ obtaining a high 
school diploma or GED tend to have favorable child impacts? For families 
with all school-age children in the 11 programs, measured child effects 
were not any more favorable in programs that did, in fact, increase the 
proportion of parents with such educational degrees than in those that 
did not have this result. 

Impacts on parents’ receipt of a high school diploma or GED do not appear to be associ- 
ated with favorable child impacts. (See Table 8.) The largest impact on receipt of these creden- 
tials was found in the Riverside HCD program, a program that did not show any favorable im- 
pacts on the child outcome measures available for the client survey sample. The other program 
that produced smaller impacts on high school diploma or GED receipt also failed to result in any 
favorable child impacts. 

• Did programs that decreased health insurance coverage tend to have unfa- 
vorable child impacts? For families with young children in the COS sam- 
ple as well as families with children of all ages in the 11 programs, there 
was not a pattern of unfavorable child impacts in the programs that de- 
creased health care coverage. 

For families in the COS sample, only two of the six programs — the Grand Rapids and 
Riverside LFA programs — decreased health care coverage as of the end of the two-year follow- 
up period. The decrease in mothers’ rating of their focal child’s health found for both the River- 
side LFA and HCD programs does not appear to be connected to this finding: Both Riverside 
programs resulted in unfavorable child health rating impacts (similar in size in the two pro- 
grams), while only the Riverside LFA program produced a decrease in family health care cover- 
age. In addition, no other focal child impacts were found for Riverside’s LFA (or HCD) program. 
For families in the COS sample, the Grand Rapids LFA program resulted in an unfavorable im- 
pact on focal children’s externalizing behavior problems, but no child health rating impacts or 
other child impacts were found. 

For families with children of all ages in the 1 1 programs, four programs — Riverside 
LFA, Portland, Columbus Integrated, and Oklahoma City — decreased health care coverage lev- 
els. Only two of these programs had any child impacts for this group of families: The Riverside 



o 



LFA program had one favorable and two unfavorable child impacts; the Columbus Integrated 
program had one favorable child impact. (These results are not shown in Table 8.) 

• Did programs that increased employment produce systematically different 
child impacts? For families with all school-age children in the 11 pro- 
grams, there is some indication that increases in employment in the first 
two years of follow-up may be associated with unfavorable child impacts, 
but this finding held true for one source of data on employment and not 
for the other. 

For families with all school-age children, the large employment impacts in the Riverside 
LFA program corresponded to unfavorable increases in school suspension rates and in attendance 
at a special class for behavioral or emotional problems, but also to a favorable decrease in grade 
repetition. (See Table 8, which presents two-year employment impacts based on Unemployment 
Insurance earnings records, as is the case throughout this document.) Employment impacts in the 
Riverside and Grand Rapids HCD programs also corresponded to unfavorable increases in at- 
tending a special class for behavioral or emotional problems and, in the case of the Grand Rapids 
program, to an unfavorable increase in the incidence of a child being removed from his or her 
mother’s care. Programs that, for families with all school-age children, did not have impacts on 
employment in follow-up years one or two had a mixture of favorable and unfavorable child im- 
pacts. 

When two-year employment impacts for families with all school-age children are based 
on parental reports of employment obtained through the client survey, the association between 
increases in employment and child impacts is no longer apparent. Using this data source, large or 
moderate employment impacts were found for the Riverside LFA, Riverside HCD, Portland, Co- 
lumbus Integrated, and Detroit programs.” The Riverside LFA program resulted in an unfavor- 
able increase in school suspension rates and a favorable decrease in grade repetition; both the 
Riverside LFA and HCD programs produced an unfavorable increase in attendance at a special 
class for behavioral or emotional problems. Child impacts for the Columbus Integrated and Port- 
land programs, however, were all favorable: in the Columbus Integrated program a decrease in 
attending a special class for behavioral or emotional problems and a decrease in attending a spe- 
cial class for learning problems, and in the Portland program a decrease in behavioral or emo- 
tional problems. The Detroit program had no child impacts. 

In sum, a clear connection between increases in employment and unfavorable child im- 
pacts is not evident. 

• Did programs that increased or reduced combined income produce system- 
atically different child impacts? For families with all school-age children in 
the 11 programs, a relationship may exist between impacts on combined 
income and child impacts. In particular, there is some evidence that de- 
creases in combined income may be related to unfavorable child impacts 
in this older-child sample. 



^*See Freedman et al., 2000, p. 193. 
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Among families with all school-age children in the client survey sample, one program — 
Portland — increased combined income in year 2 of the follow-up period; this program also in- 
creased the proportion of families with incomes at or above the poverty level. Portland had one 
statistically significant child impact, a favorable decrease in the proportion of families reporting a 
child with behavioral or emotional problems. Two programs — Riverside LFA and Grand Rap- 
ids LFA — decreased income; these two programs, as well as the Atlanta HCD and Riverside 
HCD programs, also had the effect of pushing a proportion of families more deeply into poverty 
in follow-up year 2, that is, below 50 percent of the poverty line. In the Riverside LFA program, 
three child impacts were found, two unfavorable and one favorable. The Grand Rapids LFA and 
the Riverside HCD programs each had one child impact, which was unfavorable. The Atlanta 
HCD program had no child impacts. The remaining programs had no effect on combined income 
or poverty status for this sample, and had a mixture of favorable and unfavorable child impacts. 

• Did programs with different patterns of child care use and different child 
care assistance practices have systematically different child impacts? For 
families with children of all ages in the 11 programs, the data suggest that 
child care policies may be related to child impacts for some programs. 

Patterns of child care use did not differ widely across the programs. Most programs pro- 
duced an increase in the use of paid child care (relative to control groups), but varied to the ex- 
tent that the child care increase was a fimction of increases in employment. The programs did 
differ, however, in their practices or policies concerning child care assistance. Given the impacts 
observed in most programs on the use of paid child care during employment, more information 
on the type of child care assistance offered by each program can illuminate the nature of the 
child care increases. For example, the interpretation of child care impacts similar in size for two 
programs might be different if one program paid only for licensed care while the other program 
emphasized low-cost, informal care. 

In Atlanta, Oklahoma City, Portland, and Detroit, as noted earlier, child care assistance 
was a high priority for program staff. In addition, the Atlanta and Oklahoma City programs 
would reimburse sample members only for licensed child care. For families with children of all 
ages, these five programs’ child impacts were generally favorable, although few were statistically 
significant. (These results are not shown in Table 8.) The Atlanta LFA program decreased the 
proportion of families with a child attending a special class for behavioral or emotional problems 
and the Atlanta HCD program decreased the proportion of families with a child who had recently 
repeated a grade in school. The Oklahoma City, Portland, and Detroit programs did not have any 
statistically significant child impacts for this sample. Staff in the Grand Rapids and Columbus 
programs largely expected parents to make their own child care arrangements. For families with 
children of all ages, the Grand Rapids programs did not have any child impacts, and the Colum- 
bus programs each had one impact: a favorable one in the Columbus Integrated program (a de- 
crease in attendance at a special class for learning problems) and an unfavorable one in the Co- 
lumbus Traditional program (an increase in the incidence of a child having been removed from 
the mother’s care). In Riverside, low-cost, informal child care was encouraged. In this site, two 
unfavorable child impacts were found for the LFA program: an increase in the proportion of 
families with a child who had been recently suspended from school and an increase in attendance 
at a special class for learning problems. 
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• What relationships do statistical analyses suggest? Mediational statistical 
analyses of selected impacts on young children in the COS sample suggest 
that welfare-to-work programs can affect children to the extent that they 
affect mothers’ employment and/or affect children’s home environment 
(for example, mothers’ psychological well-being and parenting). 

Five of the 14 impacts on focal children in the COS sample (which reflect the general pat- 
tern of favorable cognitive, unfavorable health, and mixed behavioral impacts) were examined in 
more detail through an analysis that attempts to identify factors that appear to statistically explain 
the relationships between outcomes. According to this nonexperimental analysis, for example, 
the Atlanta LFA program’s favorable impact on focal children’s average school readiness score 
appears to be related to the program’s favorable impacts on mothers’ employment and parenting 
skills. As another example, the Riverside programs’ unfavorable impacts on focal children’s ma- 
ternal health rating statistically appear to be related to mothers’ increase in work hours and lower 
likelihood of receiving welfare at the end of the two-year follow-up period in the Riverside LFA 
program and to mothers’ increased feelings of time stress in the Riverside HCD program. The 
Riverside LFA program’s decrease in health insurance coverage for adults and children was not 
found to be linked to the unfavorable child health rating impacts; similarly, the Riverside HCD 
program’s increase in the use of child care was not statistically related to these child impacts ei- 
ther. 



These mediational analyses also suggested that child impacts may reflect a combination 
of both favorable and unfavorable program impacts on targeted and nontargeted outcomes. The 
Atlanta LFA program’s favorable impact on focal children’s reported externalizing behavior 
problems, for example, appears to be related to the program’s favorable impact on parenting 
skills, despite two of the program’s unfavorable impacts: an increase in mothers’ time stress and 
an increase in the proportion of mothers who felt that they were “pushed” by the welfare office to 
find a job or go to school. 

• In summary, what might explain the few found child impacts? 

As discussed above, the nonexperimental approaches that have been used so far in the 
NEWWS Evaluation to attempt to explain the few found child impacts lack by necessity the rigor 
of the experimental analyses presented in the rest of the document. The two nonexperimental ap- 
proaches rest on many assumptions which may or may not be true. In addition, the two ap- 
proaches invoked different assumptions, were applied to different sets of families, and sought to 
explain impacts on different child outcome measures. As a result, they also will not necessarily 
yield the same explanations for the found child impacts. 

Nevertheless, the results for the families with all school-aged children in 11 programs 
suggest that programs that place little emphasis on helping welfare recipients obtain good child 
care or that result in decreases in family income may tend to have unfavorable impacts on chil- 
dren. (There is also some indication that increases in employment may be connected with unfa- 
vorable child effects, but this finding held true for administrative records data on employment 
and not for client survey data on employment.) More likely, these program characteristics or ef- 
fects interact with each other in particular (currently unknown) ways to affect children. Other ex- 
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amined program features or effects — whether programs were employment- or education- 
focused, the extent to which a mandatory participation requirement was enforced, increases in 
parents’ high school diploma or GED receipt, and changes in health insurance coverage — do 
not appear, by themselves, to relate to impacts on children. 

The results for the families with preschool-age children in six programs suggest that pro- 
grams might affect children to the extent that they affect mothers’ employment and/or affect 
children’s home environment (for example, mothers’ psychological well-being and parenting). 
This analysis did not find that increases in the use of child care, decreases in health insurance 
coverage, or changes in family income played a role in explaining the selected child impacts ex- 
amined. 

All of these findings suggest avenues for future research, and the longer-term impact data 
that will soon be available for both adults and children will provide a rich and more powerful 
data set with which to pursue these possible explanations of child impacts. 



IX. Conclusion 

The NEWWS Evaluation is one of the first random assignment evaluations of mandatory 
welfare-to-work programs to examine program effects on children. The analyses presented in this 
synthesis indicate that mandatory welfare-to-work programs targeted to adults, with no services 
provided directly to children, can have spillover effects on the well-being of children. An exami- 
nation of two years of follow-up found that the 1 1 programs studied in the evaluation did not 
have widespread, large, or consistent effects on the children of the parents (primarily mothers) 
required to participate in the mandatory programs. But favorable and unfavorable child impacts 
were foimd in some of the programs. Fiuther research is needed to determine the mechanisms 
through which some of the programs affected children. It is important that the parents and the 
children in the NEWWS Evaluation samples are being followed for a total of five years. Forth- 
coming analyses of five-year data will indicate whether the impacts on children observed in the 
first two years of follow-up persist, are magnified, or decline by the end of five years. In addi- 
tion, new child impacts may emerge over time. As policymakers continue to seek to both encour- 
age adult self-sufficiency and foster poor children’s well-being, these and future findings fi-om 
the NEWWS Evaluation warrant a close watch. 





Appendix A 

Two-Year Child Outcome Measures 



Child outcomes in the NEWWS Evaluation two-year follow-up are based on the two-year 
client survey and the Child Outcomes Study (COS) survey component, which was added to the 
client survey given to mothers with children aged 3 to 5 at baseline in three sites. Both compo- 
nents of the two-year follow-up included measures of children’s development in three broad ar- 
eas: behavioral and emotional adjustment, cognitive functioning and academic achievement, and 
health and safety. Some measures from each of these areas are available for all children of all 
parents in the two-year client survey sample in the seven evaluation sites; additional, more de- 
tailed measures in each area are available for randomly selected preschool-age children of the 
mothers in the COS sample (these children are identified as the “focal” children). 

Measures of Behavioral and Emotional Adjustment 

All Children 

• Behavioral and emotional problems questions. Parents were asked if any of 
their children were currently getting help for any emotional, mental, or behav- 
ioral problem and if any of them were going to a special class or school or get- 
ting special help for behavioral or emotional problems. Parents were also 
asked if, at any point since study entry, they had felt, or someone had sug- 
gested, that any of their children needed help for any emotional, mental, or 
behavioral problem. 

• Suspended or expelled from school question. Parents were asked if, since 
study entry, any of their children had ever been suspended, excluded, or ex- 
pelled from school. 

Focal Child 

• Behavior Problems Index (BPI). Parents were asked to indicate whether se- 
ries of statements are not true, sometimes true, or often true of the focal child. 
Statements describe behavior such as: the child is high strung, tense, or nerv- 
ous; the child cheats or tells lies; the child has trouble getting along with other 
children. Total BPI scores are examined, as well as subscale scores for exter- 
nalizing behavior problems (such as arguing, bullying, breaking things, lying, 
and cheating) and internalizing behavior problems (such as feeling unhappy, 
sad, depressed, unloved, or worthless). 

• Positive Behavior Scale/Social Competence Subscale (PBS/SCS). The PBS 
assesses positive social behaviors, such as self-esteem, self-control, obedience, 
and persistence. Parents were asked to indicate whether behaviors are not true, 
somewhat true, or often true of the focal child. Examples of the behavioral de- 
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scriptions in the subscale are: the child is helpful and cooperative; the child 
shows concern for other people’s feelings; the child is admired and well liked 
by other children. 

Measures of Cognitive Functioning and Academic Achievement 

All Children 

• Academic problems questions. Parents were asked if, since study entry, any 
of their children had repeated any grade for any reason and if any of them 
were currently going to a special class or school or getting special help in 
school for learning problems. 

Focal Child 

• The Bracken Basic Concept Scale/School Readiness Composite (BBCS/ 
SRC). The Bracken scale is a well-established and widely used measure of 
children’s cognitive school readiness. The 61 items used in the COS consist of 
5 subtests that assess children’s knowledge of colors, letters, numbers and 
counting, comparisons, and shapes. (The fiill scale consists of 1 1 subtests.) 

Measures of Health and Safety 

All Children 

• Accident or injury questions. Parents were asked if, since study entry, any of 
their children had had an accident, injury, or poisoning requiring a visit to a 
hospital emergency room or clinic, and if, during the same time frame, any of 
their children had been removed from their care because they could not care 
for or handle them. 

Focal Child 

• Child health rating. Parents rated the focal child’s overall health in response 
to the following question: “Would you say that your child’s health in general 
is excellent, very good, good, fair, or poor?” 
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National Evaluation of Welfare-to-Work Strategies 
Appendix B 



Table B.l 

Impacts on Average Total Welfare Payments Received in Years 1 and 2 



Site 


Program 
Group ($) 


Control 
Group ($) 


Impact ($) 


Percentage 
Change (%) 


Full sample 


Atlanta LFA 


4553 


4922 


-369 *** 


-7.5 


Grand Rapids LFA 


5944 


7347 


-1404 *** 


-19.1 


Riverside LFA 


8292 


9600 


-1308 *** 


-13.6 


Portland 


5818 


7014 


-1196 *** 


-17.1 


Atlanta HCD 


4634 


4922 


-288 *** 


-5.8 


Grand Rapids HCD 


6512 


7347 


-835 *** 


-11.4 


Riverside HCD 


9253 


10302 


-1049 *** 


-10.2 


Columbus Integrated 


4775 


5469 


-694 *** 


-12.7 


Columbus Traditional 


4939 


5469 


-530 *** 


-9.7 


Detroit 


8457 


8615 


-158 


-1.8 


Oklahoma City 


3391 


3624 


-233 *** 


-6.4 


cos sample 


Atlanta LFA 


4941 


5320 


-379 *** 


-7.1 


Grand Rapids LFA 


6194 


7742 


-1548 *** 


-20.0 


Riverside LFA 


10079 


11773 


-1694 *** 


-14.4 


Atlanta HCD 


5162 


5320 


-158 


-3.0 


Grand Rapids HCD 


7323 


7742 


-419 


-5.4 


Riverside HCD 


11350 


12095 


-745 * 


-6.2 



SOURCE: MDRC calculations from welfare records. 



NOTES: At two-tailed t-test was applied to differences between outcomes for the program 
and control groups. Statistical significance levels are indicated as: * = 10 percent; ** = 5 
percent; *** = 1 percent. 

Riverside limited enrollment in its HCD program to individuals determined by 
program regulations to need basic education, because they lacked a high school diploma 
or GED certificate, attained low scores on a reading or math test administered at program 
entry, or had limited proficiency in English. As a result, control group means differ for 
the Riverside LFA and HCD programs. 
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A Comparison of National Samples of Children 
and Control Group Children Two Years After Study Entry 



Throughout this document, comparisons are made between children of parents in control 
groups and children of parents who were subject to welfare-to-work programs. In contrast, the 
following compares the developmental status of children in national samples with children 
whose parents were in control groups. 

Behavioral and Emotional Adjustment 

The focal children in the Child Outcomes Study (COS) sample control groups, who 
were 5 to 7 years old at the two-year follow-up point, were reported by mothers to have 
more frequent behavior problems than children of the same age in a national sample. Be- 
havior Problems Index (BPI) scores indicate that control group focal children in the COS sample 
tended, on average, to have more frequent total, externalizing, and internalizing behavior prob- 
lems at the two-year follow-up point than children in the National Longitudinal Survey of 
Youth — Child Supplement (NLSY — CS) sample. 

School suspension rates, one measure of children’s behavioral and emotional ad- 
justment, were higher for children in control group families in the client survey sample 
with all school-age children at study entry than for those in a national sample of eighth 
graders. About one-fifth of all client survey sample members in the control groups reported that 
at least one of their children had been suspended from school at some point during the two-year 
follow-up period. (See the upper panel of Table D.l.) Note, however, that many of the client sur- 
vey questions applied only to children of school age. Narrowing the client survey sample to only 
those who had all school-age children at study entry, about one-quarter of the control group par- 
ents reported that at least one of their children had been suspended from school since study entry. 
(See Table 5.) In comparison, 11 percent of 1988 eighth graders nationally had ever been sus- 
pended from school.^” 

Cognitive Functioning and Academic Achievement 

The focal children in the COS sample control groups were significantly less cogni- 
tively ready for school than children of the same age in a national sample. Using age- 
standardized scores on the Bracken Basic Concept Scale/School Readiness Composite (BBCS/ 
SRC), average scores for COS control group children in Atlanta and Grand Rapids corresponded 
to the 19th percentile of the national NLSY — CS sample; the average scores for Riverside COS 
control group children corresponded to the 21st percentile. Thus, they displayed less cognitive 
readiness for school than children in a national sample. 



’®U.S. Department of Education, 1997. 
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Grade repetition rates were generally higher for children in control group families 
in the client survey sample with all school-age children at study entry than for those in a 
national sample. The national average for repeating a grade is approximately 10 percent for 
children aged 5 to 18.^' Overall, 7 to 19 percent (depending on the site) of all client survey sam- 
ple members in the control group reported that at least one of their children had repeated a grade 
in school during the two-year follow-up period. (See the upper panel of Table D.l.) The range for 
this same measure was 8 to 23 percent for those with all school-age children at baseline. (See 
Table 5.) Notably, in four of the seven evaluation sites (Atlanta, Columbus, Detroit, and Okla- 
homa City), these figures approached or exceeded 20 percent — double the national average. 

Health and Safety 

Focal children in the COS sample control groups were as likely to be rated by their 
mothers as being in very good or excellent health as children in a national sample. Between 
77 percent (Atlanta) and 82 percent (Riverside) of control group mothers in the COS sample re- 
ported that their focal child was in very good or excellent health. In comparison, in 1993, moth- 
ers of 79 percent of children aged 5 to 7 in the National Health Interview Survey sample reported 
that their children were in very good or excellent health. As noted earlier, however, parents with 
a severely ill or disabled child were generally not mandated to participate in welfare-to-work 
programs in the early to mid 1990s. While the proportion of families exempted from the partici- 
pation requirement for this reason was very small during this time period, such families would 
not have been included in the COS sample, whereas a national sample of children would include 
some severely and chronically ill children. 

Similarly, children of control group members in the client survey sample did not 
appear to be at high risk on health and safety measures. A surprisingly high proportion of all 
control group members — up to 8 percent in a site — reported that a child had been removed 
fi-om their care during the two-year follow-up period because they could not care for or handle 
the child. (See the upper panel of Appendix D.) Between 18 and 37 percent of all control group 
members reported that during the previous two years at least one of their children had had an ac- 
cident, injury, or poisoning requiring a visit to a hospital emergency room or clinic. In compari- 
son, nationally, 12 percent of children under the age of 18 in 1988 had had an accident, injury, or 
poisoning in the previous year. It is not known, however, how many of these children required 
medical treatment for these incidents.^^ 



^'U.S. Department of Education, 1997. 

“As noted earlier, this does not necessarily mean that the child was placed in the foster care system. 
Vital and Health Statistics: Health of Our Nation ’s Children, 1994. 
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Behavioral Adjustment School Progress Health and Safety 
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